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What are the relationships between policy^and 
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infora policymaking? Directors of evaluation in six state education 
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Do-novanr and Stanley A. Rumbaughr Michigan; Jaaes H. Gold, Wisconsin; 
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PREFACE 



The Research on Evaluation Program is a Northwesc Regional 
Educational Laboratory project of research/ development, cestmg, 
and training designed to create new evaluation methodologies for 
use m 8ducar.ioR. This, document: is one of a series of papers and 
reports produced- by program staff, ^visiting scholars, adjunct 
scholars, and project col-laborators — all ntemoers of a coopera^.ive 
network of colleagues working on the development of new 
methodcloQies* ^ 



0 

ion* with: 



n 



Whax are the rela tionsn :ps between policy and evaluat 
state depa rt.T.er.ts of ^ducaricn? How does policy influence what 
IS evaluated m such settings and how aces evaluation inform 
policyxakmc? Directors of evaluation m six state education 
agencies (SEAs) add:;ess tnese issuSs tr-rougn aetailed 
descriptions of their operacicns. An educational pniloscpner 
specializing m policy analysis ther provides a corirsenwarv cn the- 
crecedmg six caoers. ^ 



Nick L, Smith , Editor 
Pacer and Reoor t 5er les 
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THE RELATIONSHIPS B^<^EN POLICY 
AND EVALUATION IN STOTE DEPARTMENTS 
or EDUCATItDjJ 



Introductiof\, ' 



V/i.th the increased interest in imptcving the utilization of 

evaluation results, some researchers haW^v begun to more closely 

examine thW relationships between polidy. and evaluation. This 

topic is of particular concern to the" s&fff of the Research on 

Evaluation PrograjH because of our interl^^t in i^veloping new 

methods which increase the utilization df. evaluation. We 

* 

therefore sought to understand better ho^policy influences 
evaluation and how evaluatic^n in turn sh^pas policy. Instead of 
examining these issues through somewhat a'r|;^ficial surveys qr 
field iofeerviews/ we decided to haye the TOSt knowledgeable' 
individuals, evaluation managers themselvts, discuss their own 
insights concerning these. -matCers . ' rj ' * - * ' 

Evaluators m six state departments o^edOcation were 
commissianed to prepare reports on the in^r face, between pol^icy 
and evaluation m their setting^, l<iis do^jument .thus contains 
the. reports of the following individuals.\[\ 



Gordon Ascher , 
^ Gerald* Bracey 
f David Donovan and 

Stanl'ey Rumbaugh 

James Gold ^ 

Alfred Rasp 
. Paul Sandiffer 



Oregon Stat^ DepaS^ment of Education , 
Virginia State Dep3rtment of Education 

Mipha'gan State Department of Education ' 
Wisconsin State Department of Education 
Washfrtgtcrn State Department * of Education 
South Carolina State Department of 
Education v ^ 



To orient the writing of those authors, the following 
questions were suggested as the focus* of their repoi^ts: 

• How does policy influence evai^jation? For example: 
How does policy determine whac is .evaluated? Does 
policy somehow influence evaluation methods? How 
does policy influence tne nature and" organiza'tion of 
your .evaluation unit? To what extent do policymakers 
call on evaluators to help then with their policy 
proDlems? What policy protjlems require the use of 

- evaluation information? What percentage of 

' evaluators' tirr^e is devoted to policy questions? 

• How does^ evaluation influence policy? For example: 
To what extent is^ evaluation used in policymaking? 
Hbw IS evaluative mforinat ion comnunicated to 
policymakers? How is evaluative mforiuation used m 
. arriving at policy decisions? 

Thomas F. Green, an educational philosopher from Syracuse 

University was coramissioned to provide a general discussion of 
4 

policy formation and to comment on the six reports. After the 
authors had the opportunity to review eacn other's work at an 
invitational conference,, final drafts of the papers wfere prepared 
for inclusion here. 

Dfitrel N. Caulley, who assemDlea these materials, originally 
intended to supplement tne collection with an additional analysis 
and commentary designed to highlight the common themes of the six 
papers. Due' to a serious and lengthy illnesfe, however^^e has 
been' unable to do so. Hopefully, he will De able to return to 
that task in a few months. 

These papers present several insiders' views of the nature of 

f 

evaluation practice in -state departments of education. Their 
rich examples help illuminate the difficulties of providing 
useful evaluations in state agency settings. Readers interested 
in further discussion of these papers are 'referred to ROEP Report 
#46, '*The Context of Evaluation Practice in State Department^ of 
Education" by Nick L. Smith. * This latter report draws on these 
six papers to describe the forces that shape the practice of 
evaluation in sta'te departments. This latter paper' thus- conta ins 
additional analyses of the repoifts contained herein. 

Nick L- Smith 



THE INTERACTION BETWEEN POLICY AND EVALUATION 
WITHIN OREGON DEPARTMENT OF EDUCATION 

Gordon' Ascher , 
Associate State Super xntendent of ?ub)Jc Instryytion 
Oregon. State Department of Education 



A Conceptual Framework 

Policy a,nd Svalua^tion Defined 

It is not the purpose of this paper to develop a complete, 
conceptual framework for policy and evaluation. Its purpose,' 
rather, is to show, in a practical setting, how these two 
interact; But: it is recognized that some definition is necessary 
in order td provide a common^nderstanding which will permit 
communication between author and re^ider. For the purpose of dhis 
paper, then, the following definitions are presented: 

Policy ; Guidance provided by an organization to decision 
makers. 

Evaluation: The collection of information for use m 

; 

making decisions. 
The common factor in both definitions is "decision making.*' 

. In some manner, both policy and evaluation are used by a person 
who is forced to make a decision. How they interact w^ll^, ^of 
cours'e, vary with, the situation. But the fact that they do 

» interact is a maj.or premise here and the fact that their 
interaction occurs in the decision making* process is a second 
major premise. Therefore, we can only meaningfully define these 
terms dynamically; by considering how they fgnction jointly and 
severally in their mutual env^ironment of decision making. 

The Interaction of Policy and Evaluation 

Policy and evaluation both' serve the decision ^making 
process. They interact *with, each other in a variety of ways when 
decisions are made. ■ To illustrate, consider an unhappy consumer 
who finds* that the radio he'recently purchased do^s not 'woi:k. He 
returns to the -store seeking to return the radio. What possible 



reactions can he get from the Salesperson? The salesperson will 
either take che-radio back or not. How the* decision is made 
depends on how the store uses policy and evaluation » 

Scenario 1 : The salesperson says: "It is our policy to 
accept all (or no) returns, no questions asked." Here the 
decision appears co be ba^ed wholly on^policy. An evaluation^ of 
how the radio /funct'ions xs.not required to make the decision. We 
, should, however, suspect cnat evaluation did have some influence 
on the adoption of a policy .co accept ail returns. Probably, 
m^^et research or the storeowner's own informal collection or 
information indicaced that such a policy is, in the long run, 
good business. 

ScenariQ 2 ; The salesperson says: "It is our policy to 

decide all returns based on our evaluation of the product. We 

will have our service deoartment conduct tne evaluation', and chen 

r 

we will .decide . " The decision is based o^jj^^policy and on 

information generated through product evaluation. That is, a' 

blend of evaluation and policy. 

Scenario 3 ; The' salesperson saysj "We don't have a general 

policy on returns, but the radio is obviously defective, so you 

can return it." Having evaluated the radio's function, it 

appears that the salesperson is able to make a decision based 

only on evaluation and with no poli"cy influence. But the absence 

of policy IS illusory. Implicit in the salesperson's willingness 

to take back a' defective product is a gfblicy to be fair, or to 

please the customer, or to avoid legal action by recognizing an 
* 

impliea warranty. 

It appears from the scenarios t:|:iat no decision is made 

without the influence of both policy and evalgation. Why, then, 

* 

do our real life experiences lead us to bhink that some dec/isions 
can be made without the influence of policy; or ^^ithout the 
influence of evaluation? The answer Ij^s^ in our perception of 
the influences of policy and evaluation r^latiVe to each other on 
tne making of tne decision* Our' peroeption of the relative* 
influence of^each is determined by t?ne proximity of the in^lu^nce 
to the decision situation* A graphic presentation of this model 
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of the perceived relative influences of the making of i decision 
appears/in Figure 1: ' ^ . 

. ' Figur e 1 
HIGH= , - . • ^ * 



LOW 



NONE i 




^ ' \ ' B C D . 2 



At Point a decision maker would perceive that her decision 
«^ 

is influenced to a greater degree by policy than by evaluation. 
Similarly, at Point D she ,would perceive being influenced to a 
greater degree by evaluation than by policy. At Point C, the 
perceived influences are at ecjuipoise* Points A and 2 do not 
9xi3t in reality because every decision is influenced by both 
policy and evaluation* The misleading perceived relative 
influences are determined by the proximity of the influence to 
the immediate decision to be made. 

Point B represents Scenario li The perceived influence of 
evaluation by the salesman is .nonexistent. "Actually, the 
influence of evaluation exists but is perceived to be so small 
relative to the .influetice of policy, Lt appears nonexistent when 
the salesman says: ''It is our policy to accept all returns." 
The perception of the influence is deftermined' by the proximity of 
the influence to the decision situation. The reason it appears 



to the\salesman that evaluation is not necessary for his decision 
to take the radio back, is that he is thinking in terms of 
evalua-tion of the ^radio, the object to be returned. He does not 
perceive the influence of a more remote evaluation : The store 
ovmer's market research which led him to believe that -the general 
policy of accepting all returns is good business. So evaluation ^ 
influenced the' policy dir'ectly, but influenced the iinnediate 
decision about the ^adio only remotely and so it appeared" to the 
decision , maker that evaluation' did not influence his decision. 
Thus, it often appi^ars to us that policy tor evaluation) does not ^ 
enter into the decisions ve make. However, if we look hartd 
enough we find the remote influence of evaluation (or policy) in 
every decision we make. The model, ]ust described, can be 
suirmarized by ten "rules": 

1. ^The interaction of policy and evaluation occurs during f 

the decision making process. 

* * \ 

2. The. nature of this interaction is determined by the 

relative influence each has on the making of the 
decision. 



3'. The Perceived influence of policy and evaluation pn the 
jnakin<3 of a decision is determined by the proximity of 
the influence to the decision making situation. 

4. A decision can be, made with the direct influence of both 
policy and evaluation in the decision situation 
(Scenario 2). ' ' , 

5. A decision can be made with the direct influence of 
polity in the decision. Situation and* wi-th no direct 
influence o'f evaluation. The influence of evaluation on 
the decision exists but is remote 'to the decision 
situation. A total lack of evaluation influence is 

' illusory (Scenario 1). 

6. Similarly, a 'decision can be made with the direct 
influence of evaluation in the ^decision situation and 
with' no direct influence of policy. The influence of 



^ 1 o 



policy on the decision tkists but is remote to t}^e ^ 
decision situation., A total lack of ^polfcy influejnce is 

ill-usory fScenario 3), 

. ^ f ' ' - ' ^ 

7, Policy is cfeated or alEfirmed each time Hd^ision \is 

made. It is- common when making a decision to look^at 
^ i earlier decisions made under a similar set of ^ 

circumstances, "^If the later decision is the same aslthe 
earlier decision, the^ policy which influenced the ' 
/earlier 4ecision is affirmed. If the later decisiotu 
rejected the influence .of the policy applied in, the • 
, * earlier decision, new policy irtay be gleaned from the ne 
decision. Court decisions *f unction in this way,* • f 

8. Evaluation information is unaffected b^' a decision it 
influenced. The weight .and credrbfllit^of the data in 
future use/ however, may b^ affected by how the i3eciSion 
maker ^owed it ^t^p influencer the decision. 



9. Policy has no"influenc&=^n d^ffl^iop, making unless the 
policy is communicated and^j^nbwn- to the^ decision maker. 

10, Similarly, evaluation information has no influence on a 
decision unless the information *is communicated and is 
known to (and understood by) th^decisi'on maker. 

Policy and Evaluation in the Oregon Departingnt of Education 



The ten rules present the^e^^nce of the p<5||Licy/evaluation 
• inter^tion model. The remainder of the p^japer presents examples 
of this interaction in the Oregon Department of Education. For^ 
these examples to be understood, it is first necessary to^present 
^ ^some descrip\:ion of the Department organization and how it 
•functions. - • 



*See, e.g., Caulley, D. and Dowdy, I^ Evaluat ion case 

* ^ ' 

Ttistoriqs as a paraf^llel to legal case histories , Portland, 
Oregon: Northwest Regional Educational Laboratory. ^ 
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Organization of the Oregon Department of Education 

Figure 2 presents the Department's table of organization. 
Note t{?at.the five associate state superintendents report 
directly to' the State Superintendent of Public Instruction and 
not to the Deputy State Superintendent. This has a-n effect on 
the making of policy. Further, you will note that both the 
Superintendent and the Deputy, like each associate, manages a 
division,. Thus, they^are involved in day-to-day operations and 
do not exist in an ivory tower. This has an influence on policy 
makj.ng. The Educational Program Audit Division was es^tablished 

,by ^Superin£ender\t Duncan to separate evaluation responsibilities 
from program responsibilities. This was to relieve the 
ambivalence of program support people {e.g. Title I field liaison 

•people) who -out a great deal of energy into helping local 
district personnel develop programs to aid" children and later 
have to apply the (often antithetical) evaluation rules developed 
by feder|l agencies or others in the Department-, ' For example, 
this Division mcAiitors P.L. 94-142 commiance while the'SpeciaL 
Education Division provides technical s\jpport for program 
development in the field. Additional e'xamples are possible but 
the point is made. This innovatidd has worked to the 
satisfaction of all parties and has^ had a great d.eal of influence 
on thinking- about and doing evaluation-based decision-making 
throughout the Depar-tment. While the remaining divisions do not 
have the evaluation responsibility of the Program Audit 
Division. Yet each still must do a great deal of evaluation 
within its program support functicHis and in this case the Program 
Audit Division assists by provi^ding technical assistance. 

The State Board of Education^ a lay bo^d appointed by the 
Governor with the approval of the Senate^ has been assigned a ^ 
policy making function by the legislature. The State 
Superintendent, an elected official, carries o'ut the Board's 
policies. In^doing so he creates a great deal of policy by 
interpreting Board policy and in or^er to nill gaps in policy on 
issues not addressed by tl|^e Board. The Sqp^intendent 's powers 
are unusual in that he performs the same, func^tion as a district 
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or circuit court 'in interpreting state and federal statutes ' ^ 

concerni'ng education. Appeal from his decision is made directly 

to the appellate division of the courts. 

The policies pf the Board reside, in the administrative rules 

promulgaged by the Board. These follow a legislatively 

prescribed procedure and have the same weight as legislatively 

enacted Statutes. Board policies in many areas'have not been 

delineated, butr-niust be gleaned from the ^administrative rules. 

\ 

While the Bt)ard is curreatly considering a process to develop a 
set of policy statements, Oregon practicality dictates that 
effort prs^uce usable results. Flowery statements, or broad 
generalizations^resented by other states do little to provide 
the guidance to decision makers that the Oregon Minimum Standards 
for* Elementary and Secondary Schools (and the otn«rf sets of 
standards for other sc^hools) provide. The policies of the 
Superintendent are fou/id j.n >^ministr ative memoranda and other 
Special memoranda he'jjromulgaces and policy may be gleaned from ' 
his decisions' in controversies f(a oody tf quasi-case law) 

The»Policy Makers . ^ ^ 

While it is correct to say that the State Board of Education' 

f 

and the State Super injtendent are the policy makers identi f ied ^by 
the legislature, the reality is that many others make bolicy when 
c^rying out functions d^legafed *to them by the Board and the 
Superintendent, Therefore, bD look at policy in the .Depar tjnent , 
we must look beyond the table of <!^ganization for^a functional 
description. 

Superintendent Duncan has established a cabinet. *It consists 

f » 

of the Deputy Superintendent an^ all of the associatie 
superintendents. This cabinet meets several times each month as 
the Superintendent's Council, Note that the apostrophe in 
^^'•Superintendent 'g" is placed before the'"s" and not following ' the 
^"s." '^us, the Council is advisory to the^ Superintendent, 
Superintendent Duncan maintains his freedom to hear the Council '^s 
advice, and disregard it If a higher wisdom so dictates. The- [ 
council, hQwever, is not at all "window dressing . " The^ \ 
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discussions ar^ the arguments/put forth, are-^ serious part of the 
deci^on making process and^lso serve to keep the associates 
^current as to the Superintendent's pplicy. This policy provides 
guidance to the associates when they m^ke the many decisions 
require of them in the operation of their divisions/ The 
cduncil also serves to inf-orm the ^Superintendent of decisions 
made by associates (and other staff). These decisions have 
established policy and, since the associates acted as -the 
Superintendent ' s Agents, the decisions somewhat limit the 
Superintendent's freedom b:>*establish policy. This balance of 
policy making roles is possible because of frequent formal and 
inf ormal^intieraction among all parties involved. 

Others who influence^ policy are the special interest groups. 
The Oregon Education Association, the Oregon School Boards 
Association and the Council of Oregon School Administrators are 
just three of many. Th4 opinions of special interest groups are 
> often solicited because they offer a perspective^ which Department 
staff may not have considered. The same is true of the opinions 
pf citizens in general. 

The legislature influences policy by enacting laws which 
dictate the limitations on policy in certain areas.. More* 
directly, the legislature influences policy through the budgetary 
process. The Department's budget is presented biennially to the 
legislature. by the ^^oard.' The Board's policy to support programs 
for gifted students 'is given life by- a dollar aonount of support 
in the budget. The legislature, in nKxiifying this allocatioiv^ 
places- a limitation on the Board's application of its policy. 
One may ar^ue that the poJLicy to assist gifted students wa6 not 
affected, only the degree of suppoVt was. But the^next time the 
B6ard builds a budget, the earlier legislative action does affect 
polity, - ^ 

The Evaluators 

Jhose who hold pfyroll designations as "evaluators" reside in 
the Educational Program Audit Division. Approximately one 
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percent of the total DepartTienf operational budget , ^des ' £0 
support this division, but an estimated ten percent oi the 
operational budget is used evaluation because eve^y other 
division retains some evaluative function. So, ]ust as policy 
making does not completely reside in an, identifiable few, neither 
does evaluation, ^In fact, in reality, all-policy makers evaluate 
or use evaluation information (formally or informally) and all 
who make decisions about the conduct of evaluation are ' inf 1-aenced 
by policy and establish policy by making decisions about whab)to, 
evaluate, ^ 



The Policy/Evaluation Interaction 
in the Oregon Department of Education ^ 

The Oregon Minimum Standards 

Since decisions of all kinds on all levels of organization 

are made daily, many examples can be used *to show how policy ard 

evaluation interact in Oregon. Of all *of these possibilities, 

one has been <?hosen for disdtission 'on the basis of the fact that 

it .involves everyone in the Department, and the State Board of 

Education. The Elementary-Secondary Guide for Oregon Schools 

Part I /Oregon's Minimum Standards) has been chosen as an example 

of) (1) a policy setting process which is influenced by evaluation 

information and (2) a statement of policy which influences 

evaluation. The Minimum Standards are dis^cussed here becati'^e 

they are the Board's onrly complete statement about the Board 

policy on evaluation and about the policy/evaluatiorf 

interaction. Even though the Mi niraijn - Standards are applied at 

the local level, we use it here'for illustration <^ the policy 

that is applied at the state agency, level a^well. Xike many 

state agencies, there is not a contplete policy statement for the 

t f * t 

workings of the agency itself, tnformal transmitta3i of policy by 

the Boafd and the Superintendent i ndicate that the same, policies 

apply at? the state level, --but there 'is not the concise form to 

show the reader. 
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. »The Minimum Standards are a set. of administrative rules 
established by the State Board of Education under a legislative 
grant of authority, to do so. These have the same weight anc^ 
influence on the schcpls as do legislatively enacted statutes. 
The Board adopts many Sfits of administrative rtules but has 
chosen to designate only^pne subset the "Minimum Standards," 
While a school district is required to comply with all of the 
administrative rules, only failure to comply with a Minimum 
Standard will authomatically put into motion a proce,ss wViioh 
reguixas the district to correct its deficiency pr lose^its state 
furvJing support. This process is' the ^school staridardizatipn 
(accreditation) process administered by a seqtion of the 
Educational Prdgrara Audit Division in response to -a legislative 
.^mandate to determine that schools are meeting the standards set 
by the State^ Board. Of course, failure to comply with an 
administrative rule which is not part of the Minimum Standards 
Subset carries penalties too. Those penalties, however, are 
stated i^n'each rule or for a set of rules. But the school 
standardization process (team visits to every school in the state 
on a five-p^ear cycle) is cOTcern^d only with compliance with the 
minimum standards. • 

Let us pause and see what we can detect so f ar 'concerjiing^ the 
. policy/evaluation interaction. . * 

f — The Minimun Standards are the Board's expression of how a 
district must^p^rate to provide a quality , education to 
its students^ These are statements of policy or rules ^ 
based on policy, * 

— The legislative mandate that such policy be e^tabi-i^ed 

J was acccropanied by a jfegislative mandate that the State- * 

Superintendent of Public Instruction determine whether 
schools a^e meeting t^ese standards. This process of ? 
quality assurance is a process 'of evaluation. * 

— Thus, the legislature, , <Jfesicous of <Jaality schools, 
required clearly dei^ined policy for schools to follow and 
evaluation to make sure they do. How dc^ the legislature 
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^ ensure that the 'policy and the evaluation will interact? 
' By ^placing the Superintendent in a decision-making role. 

,*The* de'cision he must maken^ which school districts wi-ll 
continue/ to receive the state fund? provided by the 
legislature. 

The discussion of the Minimum Standards now^ branches into two 
streams. The first stream is a conside rattpn of what' the 
Standards contain because that sheds light on how the Board is 
directing the' school districts to carry out the Board's policy, 
and how evaluation ia) required to ensure that the 5olicy is 
carried out. The second stream considers how the Soard 
establishes these Standards, that is, how the Board determines 
policy and how evaluation mfomatign influences this policy 
making process.' The discussion of the first stream, .the contents 
ot the Standards, is based on the Minimum Standards currently, in 
use m Oregon aftd 'adopted by the Board on -June 23, 1976 , A 
revision process*has been underway since 1978 and a new set of 
Standards will be adopted in 1980. However, f6r our purpo^ 



nere, f ither set will do. The discussion of the second stream, 
how the Board establ?shpg standards, is a discussion of the 
revision process ;^ich- will result in the 1980 adoption. 



The Role of Policy in the Conduct of Evaluation 

The Cont'ent of the Minimum Standards 
' . J* * 

The 1976 WinimiSTj Standards are divided into twelve sections: 
Definitions . ■ These are definitions of terms used in the- 
Standards, .They a^e mote than definitions for the 
guidance o6 thQ^e inv61ved with the standards because 
these have been adopted as an administrative rule much the 
same way . that; statutes contain definitions prefaced by 
\^ - "For the purpo^ of this statutes ..." 

~ Goals. "The Board^states its goa^^s for students and, for- 
th'e process of schooling, ' 



U 



0 



O 



1 ■ 



~ Accreditation , The Board here describes in detail tts^ 
process for school standardization. That is, its process 
for responding to the legislative rn^ndate 'to evaluate 
schools fop ^ compliance with the policy wh^rh is the major 
C9ntent of the Standards, 

~ Instructional Planning . This section aontains a rule 

which requires local districts to link evaLiiation and 

policy. This rule will be discussed in more detail below, 

\ 

— Instructional I>roggams . Here are the rules which express 
the Board's policies concerning the contents of a quality 
education pfogcam. These rules constitute the Board's 
response to the legislative mandate to establish standards 
of quality, ^ ^ 

~ Administration . ^ Rules for the operation of a school 

district. This >secti on, .coupled with the sections on 
^ Student ServcSces , Staff and^Cl^ss Load , Media and 
Materials , Facilities , Safety and Auxiliary Services 
describes all policy for the operation of a local district 
which \s not contained in the Accreditation , Instructional 
Planning , and Instructional Program sections. 
The purpose of examining the conten}:s of 'the Minijnum 
Standards was to see what the Bba;:d requires of local districts 
as a mix pf policy and evaluation. It has already been stated 

the accreditation section of the standards describes how the 
State Board will 'evaluate the local districts com'pliance with the 
Board's policies, ^o the accreditation section is a view of the 
Board's policy on how the Board will assure canpliance, that is, 
how the Boar<J will use evaluation. Our purpose is little aided 
by a detailed disi:ussion of how the evaluation' is conducted. The 
rules which describe this process appear in the appendix as 
581-22-202, 204 and'206. 

We are interested, here, in how the Minimum- Standards have 
required local districts to use both policy and evaluation in the 
conduct of a quality school program. We are interested in the ^ 
policy/evaluation interaction at the local level. Taking 



selected Standards, we w^ill paraphrase oir extract excerpts for 
theisake of brevity, and use these Standards to illustrate the 

interaction. The complete text of each Standard paraphrased* 

ft 

appears in the appendix. 

, — Standard 581-22-208, Instructional Planning . Each 
. ^ district is required to establish district, progrJSn and 

course goals. These goals embody ihe stated policies 
^ about the district's educational program and the outcomes 
of that program. Onoe stated, the goals form the basis of 
the local district's evaluation of its program and 
policies about education.^ This evaluation is ^required by 
the Standard. Following the evaluation, the district is 
required ^to identify its needs by comparing assessment 
cesul'ts to its goals. The final requirement of the 
Standard is that, based on this evaluation process , "the" 
district is. Required to establish Dolicies for making 
program improvements. Thus, we see that the State Board 
has directed districts to .rely on the policy/evaluation 
interaction when planning its instructional program. 

• ~ Standard 581-22-218, Educational Programs . The Board 
requires each district to: 

(1) Identify individuals' learning strengths and 
. weaknesses; 

(2) Provide learning opportunities for students^ » ; 
responsive to their needs; - 

(3) Determine progress students make in their^ducational 
prog rant; * > ^ 

(4) Maintain student progress records and ifeport the 
information ^to patients and students (OAl^ 581-22-218). 

This Required evaluation process is intended to achieve the 
Board*s policy that districts ^provide all students opportunity 
to achieve distr ict-^do^ted learner, outcomes, ' requirements for 
graduation and personal goals through participation in , 
educational programs relevant to their needs, interests and 
abilities. (OAK 581-22-2J8) 



— Graduation Requirements ^ It is the State Board's policy 
that local boards "shall award a diplOTia upon fulfillment * 
of all 'State » and local district c^dit, competency and 
attendance requirements , (OAR 581-22-228 (1))." * 
^ Further, it is the State Board's policy that "student ^ 
transcripts shall record demonstration of minimum canpetencies ' 
necessary to: 

(1) R^ad, write, speak, listen; 

(2) 'Analyze; \ ' x ' - - 

(3) Conpute; \^ • . 

.(4) Use basic scientific and techriplogical processes; 

(5) . Dev^irop and maintain' a Wealthy ^ind and body^; . ' * 

(6) Be an informed citizen in the cc^unity, state, and 
nation; \ ' 

(7) Be an informed citizen in* interaction with environment; 

(8) Be an informed citizen on streets and highways; 
{$) Be an informed consumer of goods and ^ services; 

(10) Function within an occupation or continue education 
leading to a career (OAR 581-22-231(1))." 

It is the State Board's policy that each local district 
expresses its policy about what a competent graduate is: "The 
local board shall . . , adopt and make available to the comnunity 
minimun competencies it is willing to accept as evidence • students 
are equipped to function in the society in which they live {OAR^ 
581-22-231(2))." ^ • - ' 

The State Board policy, then, requires the local district to 
evaluate each student to see if the student has achieved 
sufficient ccxnpetence to be awarded a diploma. "Eagh local 
district enrolling students in grades 9 through 12 shall 
implement thet, competency component of its* graduation requirements 
as follows :^ 

(1) Establish minimum competencies and performance 
indicators beginning with the graduating class of 1978; 

(2) Certify attainment of* competencies necessary to read, 
write, speak, listen, analyze and compute beginning with 

. the graduating class of 1978; 



(3) Certify attainment of aj.1 competencies beginning not 
later than with the graduat^ing class of 198-1 (OAR 
• 581-22-236)." 

We have seen tihat the State Board's policy is that Oregon 
graduates be competent citizens. To achieve this, the Board 
requires each district to state its policie-s concerning conpetent 
graduates and ths Board requires an evaluationr of students as a 
vehicle for effecting its statewide policy of competent 
graduated. Or.egon diffeirs from some states in that, in Oregon, 
evaluation of the graduates is conducted locally and is uniform 
only/ to the extent that the general competency areas have been 
defined. In intent, .Oregon is no different from any state which 
establishes/ uniform exit requirsTntents and conducts a uniform 
statewide evaluation of all students* 

Additicmal examples are possible, but the intent was to show 
that it is the policy of the Oregon State Board of Education that 
(1) policies be established and corrmunicated , (2) evaluation be 
conducted to be sure ^ese policies are irmplemented , and (3) 
insures that policy an<3 evaluat-ion irrteract by requiring 
decisions (about programs and students) based on that 
interaction, Here, then, we see that the content of the minimum 
standards demonstrates that, in Oregon, evaluation is required by 
policy and policy and evaluation interact in the decision-making 
process. We now turn to the second stream of our discussion of 
the Minimum Standards to see how the development of the policy is 
affected by evaluation. 



The Role of' Evaluation in thfe Development of Policy 

We have 'seen that the Board's policy in the form of the 
!Jinimura Standards ^requires districts to use evaluation 
information when sotting policy and when making decisions. It - 
has be^ asserted that the policy about evaluation-based policy 
making in the State agency Is the same as the policy illustrated 
by the excerpts from the Minimum Standards, It can be 
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demonstrated that the Board and the Superintendent rely on ♦ 
evaluation information when establi-shing policy, 'Again, many 
illustrations are* possible , but it geems reasonable to show 'that 
the polidy statement which requires evaluation-b^sed policy 
making (the ^iniraum Standards) is, itself, policy based on • 
evaluation, ^ ' 

When the revision of the current (1976) Minimum* Standards' 
'began in 1978, the Superintendent and the Board wanted 
information 'about , the objectives of the standards and" about 
implementation, problems. In addition, they wanted information 
about the results of the impl^entation of 'the 1^76 Standards, A 
study was commissioned ,and conducted by the faculty of the 
University of Oregon. The study Was designed by the present 
author. In-order to, show the reader a real example of evaluation 
in t^ie real life of^ thte Department, the following discussion is 
based on the major portion of ^ the original paper which presented 
the design to the Superintetident and the Board. This- provides an 
insight into, the realfties of conducting evaluation' for use by 
lay policy makers. . ' ^ 



Svaluating-0p,eqot1*s Minimum Standards 



A Context in Which to .Evaluation 

Evalliation is the collection of information for the' ^^urpose 
of making de'cisions. The. evaZuation of ^ program such as the 
Oregai Miriimijn Standards consists of the collection of 
information for the*purpose of deciding whether the Standards are 
effective as they are or whether change rs" needed. The 
information to be collected relates to goals, implementation; 
procedures and observable results. None of the informa^iion, 
however, i-s useful unless there^is a well-^^efined model for 
program developnient an<J evallSation known to those involved in the 
evaluation so that the collected information can be ''inserted" in 
'the proper place in the decision making process. Simply put, if 

f . . 




we don't know. how we wish to use collected information (i.e., 
what decisiops- we wish to make) ; we will not know what 
information to collect nor will we know how to use whatever 
information He do collect. Before we can begin a discussion of 
the information we .want and how we plan to get it, musj: view 
the context within which this evaluation should be conducted. 

There are many planning models ("planning" as used here 
subsumes evaluation and the making of policy) . The model 
presented to the State Board of Education and adopted tehtatively 
as the Department's generic model appears in Figure 3.* A 
simplified. version appears in Figure 4^ This model for planning 
and evaluating programs and policy, briefly, described/ 'requires: 

1. The development of GOALS; 

2. The identification of .NEEDS (by comparing "what we want" 
(GOALS) to "what is") ; 

3. Thve fdentification of 'lONG i^GE OBJECTIVES; 

4. The identif^ication of short term PROGRAM CHANGE 
OBJECTIVES which, when achieved, will move us toward the 
achievement of the GOALS; 

5. TJie development of a PLAN to achieve some of the PROGRAM 
CHANGE OBJECTIVES; 

6. The implementation 'and eventual evaluation of the 
effectiveness of the PLAN (an effort which seeks th^ 
answer to the question "are the goals of the PLAN being 
achieved?"); and finally; ^ 

7» We ask the questions "are the PROGRAM CHANGE OBJECTIVES, 
LONG RANGE OBJECTIVES and original NEEDS being met by 
this plan?"' Following this effort we -make decisions 
about the efficacy of the PLAN the policies. 



If an evalUator 'who knew nothing of the history of the 
development of the Minimum Standards were given this generic 



, _*The Oregon Planning Model was developed jointly by Gordon 
Archer, Robert Clemmer ^nd Donald Egge , all of the 'cXregon State 
Departmeht%f Education. ^ 
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model and ^a copy of the Minimum Standards, he/she would probably 
assume that prior to the adoption of the Standards the Board: 

1. Established GOAf^ which described t;he Board's philosophy 
* arnd what it Hoped to accomplish "through a variety of"^ 

means ; 

2. Used assessment procedures to collect information which, 
, when "canpared to .the GOALS, identified the NEEDS {i.e., 

the Board used the discrepancies b^etween what it w^inted 
and "what was" as statements of NEEDS) ; 

3. Identified some LONG RANGE OBJECTIVES; ^ 

4. ^ Identified some short term PROGRAM CHANGE OBJECTIVES 

which, i-f accomplished, would lead the Board closer to 
the achievement of its GOALS; 
A 5. Developed a PLAN (the Minima Standards) which it hooed 
wogld achieve some of the PROGRAM CHANGE OBJECTIVES, and 
6. Implemented the PLAN with an evaluation design to answer 
the question "Are the go^ls of the PLAN being achieved?" 
All of this so tj}at, within the context of the generic 
model, the Board could appraise' how well the identified 
NEEDS were being met\ 
The evaluator'6 assumptions would be je^asonable but 
erroneous. In fact, prior to the implementation of bhe Mil^imum 
Standards, .the Board di^ not^ establish GOALS, identity ^EEDS or 
identify LONG RANGE OB^gKtVES or PROGRAM CHANGE OBJECTIVES'. The 
Board did develop a PLAN, albeit in the absence of a sound 
context, but neglected td develop 'an evaluation component for the 
PLAN. As a result, the Board cannot, it would seem, readily 
'conduct a proper evaluation, after the fact. But/almost all of 
the p^rts of the generic model do exist and were attended to but 
these parts were not brought together as a cohesive whole. It is 
possible to do so because of the vantage point presented by the 
passage of time and because the work done to date has been of 
high quality-. ^ 

We can conduct a proper evaluation because it is possible to ^ 
reconstruct some of the missing links in the generic model. The 



evaluation a'nswers four basic Questions, ''Are the P"RDGRAM CHANGE 
OBJECTIVES ^being achieved?'\- "Are the LONG RANGE OBJECTIVES being 
achieved?"; and "Are the identified NEEDS being met?" 



The Need for Reconstruction 

Considering the Question "Are the PROGRAM CHANGE OBJECTr/ES 
being achieved?" To ans-y^r this question we need to know what 
the PFDGRAM CHAi^GE OBJECTIVES are* that we are trying to achieve 
through the Implenientation of the Minimum Standards ^the PLAN). 
Further, to evaluate the effectiveness of the'^PLAN we need to 
know the goals of the PLAN itself, and the^ performance 
indicators, measures and standards related to each of the goals 
of the PLAN. Of all of th^se, we have only the PLAN, We must 
identify the PROGRAM CHANGE' OBJEVTI^/ES , the goals of the PLA^ and 
the performance indicators, measures and standards related^to 
each goal. 

Once we have answered this question we can ask whether the 
LONG PJ^GS OBJ£CTX^/ES and NEEDS are being met. To do so, 
however, we must identify the LONG RANGE OBJECTIVES and NEEDS and 
to do that we must first identify the GOALS and the assessment 
information which enabled us to determine needs relative to the 
G0AL5.^ ' , * ■ 

Upon accc«nplishing all of this, we will have the elements of 
a Drooer evaluation. - * 



A Plan for Conducting the Evaluation 

Part I. Evaluating the Minimum Standards Out of Context 

This section presents a plan ^r answering the question, 

i 

"Have the goals of the Minimum Standards PLAN been achieved?" As 



*We could use the past tense but we want our r^ults to be 
useful now so we will forget historical objectives and work with 
those currently in place • 

\ ' c I 



\ 

point out $bove, this question is important but it is only one of 
our iin,^rtant questions. ^Trying to answer the remaining rthree 
question^** requires placing the answer to the f-irst question 
within th^ context'of the Board's generic planning model. 
Evaluation of the'^ Minimum Standards within that context is the 
subject matter of Part ii. . 

To determine whether the goals of the Minimum Standards PLAN 
have been achieved we must first identify th# goals and then . 
identify performance indicators for each goal* Following that, 
we must identify appropriate measures and standards to determine 
whether the performance indicators_ were achieved and then make 
inferences about the achievement o^the goals. We can define 
twelve tasks: 

1. Identify the goals of the Minimum Standards PLAN. 

2. For each ^oal, identify one or more perfojs^^nce 
iTx3icators. ^ 

3. For each performance indicator, identify one or more 

^ appropriate measures and performance standards which will 
• be used to collect and analyze information about the ^ 
achievement of the performance indicator. 

4. Collect and^analyze data, 

5. Frcsn information about performance indicators, infer 
about the achievement of each goal. 

6. Identify 'the plan for implementation of the Minimum * 
Standards.^ ^ 

7. Identify the implonentaticxi pl^ln goals. 

8. For each goal, identify- one or more performance 
indicators. 

r 

9. For each perfformance indicator, identify one or more 
appropriate measures and performance standards which will 
be us^d to collect and analyze information about the 
achievement of that performance indicator. 

10. Collect and analyze data. 



**The remai-ning three questions are: (1) "Have the PROGRAM 
CHANGE CBJECTIVES been achieved?**, (2) **Have the LONG RANGE 
OBJECTIVES been achievedTy*, and (3) "Are the NEEDS being met?" 
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, 11. From information about performance indicators, infer 
'about the achievement of each implementation goal, 
12. Produce a report, on the achievement of program and . 
implementation goarls. 

t 

Part 11, Evaluating the Minimum Standards Within the Context of 
the Generic Plannina Model 

. n 

Here we attempt to answer the questions "Have the LONG RANGE 
OBJECTIVES been achieved?" '*Have the PROGRAM CHANGE OBJECTIVES 
been achieved?" and "Are the NEEDS being met?" To answer these 
questions we m^st/first have the answer to the question posed in 
Part I "Have the goals of the minimum Standards PLAN been 
achieved?" Before we can answer these th%ee questions we must 
also fill in the missing parts of the generic planning model. 
These missing parts are the otate Board of Education GOALS, 
identified NEEDS LONG RANGE CBJECTIVES and PROGRAM CHANGE 
CBJECTI^/ES. We can define six tasks: 

1. Identify the State Board of Education GOALS. 

2. Identify the NEEDS. 

3. Identify the appropriate LONG RANGE OBJECTIVES. 

4. Identify the appropriate PROGRAM CHANGE OBJECTIVES. 

5. Apply the data garnered from the evaluation of the 
Minimum Standards PLAN (Part I, above) to: 

a. determine whether the PRDGRAM^HANGE OBJECTIVES were 
achieved/ 

b. determine whether the LONG RANGE OBJECTI^/ES were 
achieved, 

c. deterjnine whether the NEEDS were met, 

d. infer about the achievement of GOALS, 

6. Produce report bn the achievement of LONG RANGE 
OBJECTIVES, PROGRAM CHANGE' OBJECTIVES and the meeting of NEEDS. 

( 
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The preliminary portion of the evaluation study has been- ' 
conducted. The results of- an extensive survey of several 
school-based populations are available. These results are 
cur.rently being used by the Superintendent and the Board in the 
setting of new policy, ^ «-Jv 



Summary 

This paper has explored a conceptual framework for the 
relationship between policy and evaluation and has provided 
examples of the policy/evaluation interaction in the Oregon State 
Departinent of Education, The author has attempted to s}j^ that 
policy makers are very much dependent on evaluation information 
and, in fact, are requii^Nto be so dependent by the po^^ies of 
the State Superintendent of Public Instruction and the'^StcCte 
Board of Education, In addition, it has been shown that 
evaluators and the nature of evaluation are guided by the 
Department *js written policy. 



THE RELATIONSHIP BETWEEN EVALUATION AND POLICY . 
IN THE VIRGINIA STATE DEPARTMENT OF EDUCATION 
; . By , > . 

.■) ■ Gerald W: Bracey, Director 

Research, Evaluation and Testing 
^ Virginia Department of Education 

• 

Some General Comments .and Definitions^ Issues , r 

Judging from the literature I have read, ^evaluation and policy appear 
to have bees only tangentially or occasionally related at- either state or 
federal or even local levels.' Where they have been related, the relation- 
ship has generally been unidirectional with policy having a significant 
bearing on evaluation, but not vice versa, ^xplanatipns for this phenomr 
enon vary. Fox (1977), reviewing 10 years of evaluation of Teacher Corps 
concluded that^the problem lay pri^narily with the.methods chosen. Accord- 
ing to Fox the "Standard Experimental Design" as defindfd by Astin and 
Panos (1971) was often used and was invariably inappropriate. Fox recom^' 
mendell to evaluators the "model" by Parlett and Hamilton (1976) described- 
as "Evaluation as Illumination". Certainly evaluators have been misled by 
certain methods. However, in this- case, I feel that a careful reading of 
Fox* paper leads qrie to conclude that the policy changes affecting by 
Teacher Corps would have taken place no matter what evaluation model bad 
been chosen. Policy changes were determined by other factors and the 
outcomes of evaluations were simply irrelevant to such decisions. Indeed, 
House (1979), has argued thk the Follow Through Program in particular and 
f-ederal programs in general h^ve^ been evialuated in such a way that, the 
evaluations are bound to be narrow, tri^/ial and hence rrelevant. Many 
would argu6,-I included, thati:he current Title I evaluations- wi'll be 
another case of a disfunction and disjunction between ^valuation findings 
and pol icy changes. 



All of this prologue is simply to document that the relationship 
between evaluation and policy has not been a happy one, at least as viewed 
from the perspective of the evaluators. Mkin and Oalllak (1979) recently 
lamented: \ ' 

There have been great^ hopes for evaluation / not only 
^ among evaluators themselves, but also among m^ny other 
educators, elected officials and the public. Yet these 
hopes have* dimmed. It wasHioped that evaluation infor- 
mation would.^help planners, administrators and policy 
makers both by improving individual programs and by 



Cautionary note: While readers may experience difficulty With this, 
paper for a number of reasons ' solely the fault of the author, ^ readers 
addicted to the words "Implement" and "Impact" will experience additional 
difficulties as these words occur nowhere In the text in either noun, 
verb, or adjective form. 
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^. aiding in choices among programs. The reality we are 
Y told by almost all observers, is that evaluation has 

had litt>e influertce on educational decision making, 
and evaluation infoonation is largely ignored. 

Although most ^valuators (and even policy makers) reading that state- 
ment would probably. give general asserrt, the sirStement by itself has 
several problems. It does not delineate types of evaluation or* define 
pol'icy. A necessarV first step in understanding the role between "evalu- 
ation\and "policy" i\i the Department of Education or anywhere else is 
to define both evaluation and policy. 

Atone level^ any decision that determines policyjs based in part on 
iDformation that''could be called evaluative, HoweyerTsuch information 
CQuld be (and often is) derived largely frofn personal experience, Pipho 
(1978) reported that 80% of all legislation introduced with respect to 
Minimum Competency Testing Priwrams -resulted from the irrmediate experience 
of a legislator with either hil own family'or that of neighbors or rela- 
tives. This datum alone shoulc\give some indication of the relative force 
of large evaluation ^oje^s on >slicy decision's, ^ While these legislators 
are clearly evaluatin^-4tTfdnnat;KQn^^^^ using personal experiences as a 
source of evaluative information, shch experiences do not constitute^- an 
adequate definitiqn for evaluation for the purposes of this paper,* 

Jcfr the pur^poses of this paper, "evaluation" or "evaluative informa- 
tion" will refer to any of the ten categories of studies classified by 
Webster and Stuff lebeam .(1978) as auasi evaluation or true evakation."^ 
While it would be fascinating to include a discussion of what those author 
refer t'o as pseudo-#/e^uation"propagandist.ic studies designed to support 
predeterniined policies— such discussion would be difficult as, by defini- 
tion, the goals are predetermined and part of the relevant information is 
either not collected or deliberately cpncealed by the perpetrators of the 
study. Note- that one of the ten cate^ries itself . is 'labeled "policy 
studies" which foreshadows what wilj/become clear-^that the relationship 
between the two concepts is sometiifies intricate and simultaneously elusive 

If one can obtain a frame of reference for "evaluation" by noting 
a single synoptic paper, such is not the case with "Policy". Attempts 
by the author to obtain concre'te, short definitions of policy have failed. 



• ♦While personal experience does not constitute "evaluation," an evalu- 
ator with an interest in infl(^encing policy would be well advised to see 
that his information became a part of the "personal experience" of the 
policy maker. Constraints on evaluators doing this will be discussed latei 

^hese categories are objectives based studies^ -accountabi 1 ity 
studies, experimental research studies, testing programs, management infor- 
mation systems, accreditation/certification studies, policy studies, deci- 
sion oriented studies, consumer oriented studies, client oriented studies, 
and connoisseur based studies. Although Stufflebeam and Webster present 
them as "types" the author does not presume ^ them to be mutually exclusive. 
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The genera-1 thrust of the answers has been "I can't define policy but I 
win know one when I see one". Dictionaries are no help and articles 
about policy provide 0 1ttl e guidance because the word is used in so many 
contexts. It^is endowed , as philosophers of science put U, with a great 
deal of "surplus'meaning". Generally, people respond-1-ng to the .question , 
"What is Policy"? discussed an overall 1 philosophy, ideology, or plan of 
action, sometimes with clear budgetary implications, sometimes without. 

After a number of these discussions I imagined a continuum running 
from broad-scoped statements*^of "policy" at one end to specific state- 
ments of administrative Procedures at the other. I was, however^ unable 
to determine any point along that continuum where policy clearly became 
procedure or vice versa. Indeed, it will be seen during this paper that 
one source of difficulty in relating policy and evaluation is a conf^jsion 
among all the actors involved as to where policy ends and administration 
begins. A result of this confusion is that administrative procedures, 
best left as such, get elevated to the level of pol icy' maicing, with a 
resultant territorial intrusion by one group on the other. Similarly, 
policies get enacted with no clear procedural implications. Almost any. 
course of action can be restated as policy. "It is the policy of the* 
United States to contain communism wherever it should appear". Few would 
argue that this is. not a policy of the U.S. government, but its procedural 
implications are by no means clear; the possible translative strategies 
range from providing effective demonstrations of the superiority of a 
capitalist system to providing assistance to any government promising to* 
fight against communist activity in that country to physical obliteration 
of all nations professing thetr^selves to have communist governments. 

In other cases to be discussed, it appears that pojicy statements 
often make an appearance to justify administrative procedures already in 
place. Unless, of course, those procedures themselves are under attack. 

Finally, in some casas the policy statement is redundant with or a 
gratuitous addition to a procedure. The ^78-80 Standards of Quality for 
Public' Schools in Virginia, tote di scussed'' in detail below, illustrate 
such a case of redundancy. One part of one standard states, "It is the 
policy of the Commonwea.l th that the awarding of a high school diploma 
shall be based upon achievement". There follows the standard that puts 
IntQ place Virginia's graduation competency testing program. The policy 
statement is unnecessary, is unattached in any causal way to' the testing 
program. In recognition of this fact, some legislators uns'ucces^fully 
attempted to have the policy sentence removed before the standard was 
enacted into law. . ' 

It is comforting, if not illuminating to note that others have . 
struggled with definitions of policy issues and it is worthwhile to 
examine the struggles of other^wlth policy. 

Berl-ak (1970) pointed out that evaluators needed to know whether 
they were operating^ in an area of programmatic 1mpact*or policy impact* 
and ^act accordingly. For Berlak, a policy issue has four criterial attri- 
butesb' 

'■^These are phrases from Berlak and do not violate the promise of page 1. 
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1. It directly /S4:/indirectly' alters the power relationship between 
the citizens and th'fe state. 

2. It affects .immediately ar in the long run the status a person 
has and the power he can exercise within the social system. 

3. It increases or decreases political or sicial tensions as a 
primary outcome. 

4. It alters the self concept of the individual. 

While not a model of clarity, nor in several instances easily amenable 
to empirical tests, these considerations are an injprovement over the 
classical political definition of policy as the "application of reason 
and evidence to choose among program alternatives". Perhaps the .best . 
discussion -of policy is by Mann (1975) who chooses like Berlak not to 
define policy but to describe policy issues in education again in terms 
of criterial attributes. For Mann, policy issues have five characteristics 

1. Policy problems are public. This might at first seent unnecessary 
to state as we are dealing with public education and hence all 
things are in public domain, but in fact they are not. Certain 
"policy" discussions revolve aroung what is the proper goal of 
^ public education. Which of^ the desired outcomes of childhood 
are the domain of the school and which are those of family and 
other institutions? Certainly much heat, if npt light has been 
generated over where the responsibility of schools* ends. 

Z, Policy problems have important consequences. They increase ten- 
sions among political groups and their resolution directly affects 
the lives of a large number of people or dc small number of 
people in large ways. 

3. Policy problems ar$ complex. They have political, economic, 
moral, dimensions which of course, do rfot operate ind'ependently 
but interactively. 

4. Policy problems involve large amounts of uncertainty. This 
almost follows from #3 above. ,If the state decides to allocate 
more funds to school districts with low scoring cTiildren, in 
order to hire more teachers what will be the outcogne? .Clearly 
this cannot be known in advance although various "scenarios" 
can be depicted with greater or lesser sophistication. 

5. Policy problefT]^ are viewed differently by those with different 
interests or Ideologies. Again this appears trivial or at least 
axiomatic but it is important to state. If all people agree on 
what is to be done, then it is no longer a policy problem. And, 
in terms of this paper, the fact that people disagree has direct 
bearing on how, when or if evaluative information will be used. 

Attribute five, even if axiomatic, is important to the thrust and 
tone of this paper. I take it as axiomatit that readers will not be inter- 
ested in the effect of evaluation on policy unless that evaluation contri- 
butes to the resolution of a policy issue. 
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The Structural Context . 

There are, fundamentally, three policymaking agencies in the area of 
education: 

1. The Department of Education, a part of the executive branch of 
the government. 

2. The State Board of Education, appointed at the pleasure of the 
Governor and operating through the Department, but often quite 
independent of it. 

3. The legislature. ' 

Policy matters may originate 'with any of the three. Generally, any 
policy matter originated in the Department is brought before the Board 
for approval, and the Board, seeking to take a stance on a policy matter,, 
may instruct the Department to provide the information necessary to making 
decisions. 

I -will show in some detail in the 'next section that most major policy 
issues either enter the arena via the constitutionally required "Standards 
of Quality for Public Schools'^ also to^ be discussed below, or such policy 
issues come to rest there. In general,' the Standards of Quality (SOQ) a^fc 
drafted each biennium by the Department staff^"^ approved with revisions b^m 
the State Board and approved with further revi^ons by the legislature. 
The legislature enacts the SOQ into law. Ltis thus obvious that the ^ 
legislature is the final' arbiter of educa^onal policies contained in the 
SOQ no matter where the policy originat/d (unless the entire act is vetoed 
by the Governor). Again, as I will discuss in both the Historical Context 
and the Case History, the legislature in recent years has been very active 
in resolving new policy issues and reacting to policy matters brought to 
them by the Board and the Department. 



The Governor may, of course, ir^dependently formulate policy through 
his office or through those of- the Secretary of Education or the Super- 
intendent of Public Education. In recent years, where Governors have 
acted independently, the target of their actions has been largely the 
domain of post-secondary educatio/i, not elementary or secondary public . 
schooling. The Superintendent of Pubjic Instruction and the members of 
the Board are all appointed by the"^ 'Governor, subject to approval by the 
legislature. According to the constitution* the Superintendent's term is 
coincident with tha.t of the appointing Gover;nor; •In fact, no Superintend 
dervMlas gone out of office with the appointing Governor. It might ±ie 
noted the Virginia governors may hold two f<5ur-year terms, but not sue- 
sessively* The nine members of the State Board are appointed for a four 
year term anjl my succeed themselves once. Terms are staggered such that 
no more than one vacancy occurs at a time. In the two and a half years 
that I have been with the Department, five of the nine members have been 
succeeded, although one of the five resigned due to. the press of\ther ^ 
duties as the mayor of a large city. 
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Th^.^-eT^ti^ Board have been appointed by Republican Governors 

although approved by Democratically controlled legillatures. It is some- 
times alleged that the differing views taken by the Board and the l^egisla- 
ture' on matters in recent years are a reflection of this differing parti- 
saViship affiliation. That the legislature fights some of its battles with 
Governors throt^gh the vehicle of overriding the Board. Not only is the 
truth of this statement not ver.ifiablej^the al lection is not readily 
, apparent in public exchanges?. I was t6ld by several people in all serious- 
ness that the dififerences produced by p>urti3an affiliation would^e 
obliterated by the communality engendered by Virginia' s •longstandT)ig 
coimitmeht to conservative tradition. - 

Final'ly, the media, especially the print nfeiHia, have shown no ren^c- 
tance to propose or react to policy issues relating to public educatKm. 
This might be expected by the description of policy issues in the.pr^eding 
-section. As pertains to evaluation, the pr^^has'been particuUrlyNicti ve 
in commenting on the results of state manda ^ y ^l esting programs 'and trends 
in test results both in the state and nation. As will be seen, testing 
has come to constitute the of evaluative information about public 
schooling at the state, level. * > ' j 

'\y ^ ^ ' 

Figure 1 shows the structure^ of the Department of Education as of 
December 1979. Only those areas outside of the Director of Research, 
Evaluation -and Teeing which are also concerned in significant ways with 
evaluation are laWlled.^ Both the components pf the Division of Research, 
Evaluation and Testing (DRET) and the way in which information flows from 
^^j^^o other parts j)f tbe agency meri.tj comment.- 

It is an administrative regulation of the Superintendent that any 
information intended for the State Board rjusj go through the Superinten- 
dent's office. While the previous SO^intendent delegated much autonomy 
to the assistant superintendent for ^t^. flow of $uch information, the 
incumbent personally "signs, off" bn alljsuch information. The importance^ 
of the quality of >nfoniiation flow from lower to upper levels in the ' 
hierarchy, is manifest. ' - 

Similarly, it is i regulation that contacts between the Department, 
officially a part- of the executive branch, and the legislature cannot / ^ 
be initiated by Department membersf The Assistant Superintendent for 
Administrative "Field Services serves as the official liaison between the 
Department and the legislature. Any information thought important to the 
legislature must be funneled through him. Legislatoi^ 'may contact mem- 
bers of the department informally for information or request them to ' 
testify before the various OKinmittees and subcointiittees of either house. ^ 
A, brief report of all such contacts initiated by legislator's in this 
way must be filed with the Assistant Superintendent for Admini'strative 
Field Services. 

Virginia's DRET congtons several functions which mi^ght^rrot expected 
to be found in a so-namefflpoup and does not contain several other func- 
tions which one might, exp^c^it to have. - Notably it ha^ r^responsibility . 
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for the evaluations of Title I, special education, or vocational education. 
While this leads to administrative akwardness and some redundancy in 
efforts (children tested for Title I are retested under the state program, 
but the magnitude of any practice or regression effects is unknown) it has 
little real impact on policy at the state level: most of the above men- 
tioned programs are constrained in tenns of evaluation by federal, not * 
sta1;fi policies. Virginia has a long history of discordant relations with 
Washington and One often gets the impression that; where feiteral funds are 
involved, the Department prefers the offices handling* these tainted monies 
to 'be as autonohious as possible,* 

This general feeling does not hold true entirely,, as witness -ihe place- 
ment of the Title IV-C office with the DRET area. While arguments could 
, be made for its being elsewhere (and until 1977 it was with the Special 
Assistant for Federal Programs), coupled with the Pilot Studies program,' 
which is a 'state funded, innovative R&D effort, its placement with the 
Program Development sphere and R & D unit also makes sense. 

What is most noteworthy and important is that while there is a research 
staff designated as such and s testing staff designated as such; there is 
no evaluation staff designated as such. While accreditation faWs into a 
valid category of evaluative infoiprotion, the metKods used in Virginia do 
not at the inoment conform to the typical methods citgd b^ .Webster a#id 
Stufflebeam for these operations , notably selKstudy and team visits. 

Accreditation is,>ased, largely on self-report on a questionnaire aug- 
mented by site visits. There is*a secondary school evaly^ation designation, s 
with one person in this role who organizes those teams which make site 
visits following a period of self-study by the local .agencies. There are 
reconmendatiohs made from these comittees but there- is no follow-up to 
determine if the recommendations have been acted on and in any case no 
sanctions^ are imposed if proWem^ are noted but recommendations are not 
followed. Virginia is one of few' states which separates the accreditation 
and evaluation process for schools; the evaluation process is voluntary-- 
" it must be requested by the local superintendent. 

The structure oK^he_DB£T leaves little or no opportunity for what is 
usually thought of as program evaluation. The research staff have skills 
appropriate tb such evaluation, but for the most part their energies are 
devoted to either thp periodic surveys ^that the defpartment conducts Or to 
ass>sting the testir^ service nith the myriad of programs which have been 
mandated by the legislature ^^the past three years .(^ y 

The lack of evaluation staff constrains the DR£t in two ways: It 
must either rely on. the assistance*bf people from outside the division not 
^ well trained in evaluative techniques or more likely, it muH turn to an 
/ outside' agency such as a umversitY, .contracted for a particular task. 
\While Such contracts have advantages as well as^ disadvanj:ages , the chief 



*The Superintendent from T975-78 was previously local superintend 
""dent; of the only division in the State that did not have a Title I program. 
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disadvantages .are ^that university faculties are often not aware of the 
information /needs of the DeDartment, try unsuccessfully to fit the 
problefT) Into the paradigm of academic research, and cannot be "on site" 
^ften enough to render assistance at its most timely occasion. . 

The Historical Context . 1 

As indi<fated in the previous section, the use of evaluation is 
affected significantly by the structural context of policy making. It 
would be^a mistake, however, to view tnis context as static. Indeed 
the purpose of this section* is to treat that structural context as a 
dynamic, flvid one^^and-show how it has changed over the past decade. 
The primary oiB'^rTsof focusing on the historical changes ^ill be by 
following the .evolution of the Standards of Quality for Public Schools 
in Virginia and the evolution of the position of Director of Researcn, 
Evaluation and Testing as reflected, in part, by changes ii^ the job 
description for that position. 

While any starting date for a "history" would be -arbitrary, the 
year 1971 seems appropriate to demarcate a formal change in thinking 
about education. By 1971, the policy of massive resistance to school 
integration had in itsel^ been largely abandoned. No issue had domin- 
ated public education in Virginia ^ite the .way that th^ ramif icatidns " 
from Brown vs. The Board of Education had ("Thi-s will keep us in power 
for at least twenty-five years", -said one legislator in the 1950s, clearly 
seeing how long the issue would be in the forefront), but in 1971 the 
d^oate revolved not around whether, but how (i.e. through voluntary 
efforts or court-ordered busing). More importantly, in 1971, the Cofmnon- • 
wealth of Virginia approved a new constitution for .the state. A part 
of that constitution reads as follows: 

Standards of quality for the several school divis^ions 
shall be determined and prescribed from time to time . ^ 
by the Board of Education, subject to revision only ' _ 
by the General Assembly* 

The General Assembly shall determine the manner in which ^ 
funds are to be provided for the cost of maintaining an 
^ educational program meeting the prescribed standards of 
quality and shall -provide for the apportionment of the 
cost of such program between the Commonwealth and the 
local units of government comprising such school divisions. 
Each unit of locaT government shall provide its portion 
of such costs by local' taxes or from other available 
funds. ART/VIII #2. 

The Standards of Quality (SOQ) hav^ come in the last eight years to 
be the final resting place of al^ policy issues where same state agency 
1s the initiator, of policy. Almost all najor -^ol^cy matters have either 
begun with changes in the SOQ, or if begun elsewhere (as, say, a change 
in the standards for accred-tihg secondary schools), have eventueMy 
made their way into this document. Similarly, most of the debates over 
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policy issues at the- state level^ can be found reflected in the chajiges of 
the SOQ over. time and in the manner in which they were changed by the 
respective bodies who wrilft and rewrite them. 

\ The original SOQ for 1972-74* were performance-oriented with^a number 
0*1 standards being expressed in quantified terms relating to outcomes for 
bqth schools and individuals. Most of these outcomes were countable and 
stated 1n terms of expected changes in the- future. For example: 

.At least 45>000 five-year-old children In the State 
will be enrolled in kindergarten (26,500 in 1969-70). 

Only one standard actually dealt wilh learner outcomes /and that stan- 
dard, meaningless on its face was to cause considerable controversy and 
lead to changes in the SOQ. This standard and the controversy it produced 
will be discussed as part of the illustrative Case History. 

In 1973> the position of Directo'^of Program Evaluation (later to 
'become Director of Research, Evaluation and Testing), was created and 
carried, in part the following job description: 

1. To provide leadership in evaluating State objectives (Purposes of 
education adopted by the State Board of Education), programs 
(including ^tandards-af-^uaVity)^, "and-^tudent progress: " 

2. To develop a training program for the State Department of Edu- 
cation staff and for school division personnel in translating^ 
state objectives into learner-oriented objectives » many of 
which should be measurable. " ^ 

3. To develop, with the assistance of consultants and a representa- 
tive committee, the criteria needed by school divisions to evalu- 
ate their own programs, organization and procedures, reporting 
and progress— especially the progress in student learning. 

4. To assist schools and school divisions in making realistic ^ 
evaluations and reports to the public. 

5. To encourage and assist institutions of higher learning to evalu- 
ate their programs for pre-service and in-service preparation -of 
teachers. 

«^ * « 

This broad, wide-ranging set of duties was never realized. The Pro- 
gram Evaluation staff consisted at first of only the Director and one 
other professional and later a second professional staff member was added. 
One reason for this lack of support was that the St^nd^rds of Qua,lity 
themselves were being revised substantially for the 1974-76 version in a^ 
way so as to downplay the need for evaluation. In this version, the per- 
formance standards which had led off the list of original standards were 
eliminated. The general goals of public education were stated and five 
modified performance standards were listed as "objectives" with the note 

that "school divisions may wish to establish additional specific objectives 
to receive priority (luring during the biennium". 

♦ ' g 

♦Virginia^perates most of its programs , on the basis of biennial plans 
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The bulk of the standards, listed as such, were ten prescriptive 
statements categorizable^as "input" standards. They Specified that 
schools were to' provide kindergarten, special education, gifted and 
talented education, a certain number of professional .staff for each 
certain number of students, etc. * 

This shift away from the outcomes of schooling back to the provision - 
of goods and services--inputs--did not sit well with the legislature. 
According to persons k interviewed who were around at this time, the 
legislature felt thai the intention of the SOQ section of the 1971 con- 
stitution had never been properly enacted by the State Board of Educa- 
tion. In any case, the 1974 session of the legislature noting that 
there had been "discission both within and without the General Assembly 
as to wHA^f is appropriate to be included in the Standards of Quality...," 
created a joint subcommittee to study the SOQ. Formed in September 197^, 
the committee delivered its final report in December, 1975, just before 
the General Assembly- convened for its 1976 session. 

The- words of the report itself reveal as well as anything can, the 
scooe and philosophy of the subconmittee's study: 

Rather than' confining its work on*ly t^ the language of 
the standards, the joint subcommittee has sought to review 
comprehensively all aspects of publicly financed education 
in Virginia. ' 

The outcome of this review, again in the words' of the committee was: 

To a great extent, the Joint Subcomnittee's revision /jof-* 
the Standards of Qaulit^ has been based on the concept 
that the quality of education is measured ultimately by 
what students have learned (output) rather than the quant- 
ity /)r quality of resources devoted to education (input). 
Whereas some standards must be oriented towards input, the 
' greater' emphasis should be, in the opinion? of the Joint 
Subcommittee, on output.^ 

Thus very clearly and dramatically did the General Assembly, on 
adopting the SOQ as revised by the Joint Subcormiittee change the thrust 
of educational policy away from the tradTtional goal of providing goods 
and services to measuring outcomes. In fact, many people outside of 
the General Assembly felt that the legislature had not revised the SOQ 
so much as they had actually rewritten them, thereby overstepping their 
constitutional powers. According to these same observers, the Board 
decided that any constitutional challenge could result, at best, in 
winning a battle but certainly losing the war; accordingly, no such 
challenge was made. 

Two other quotes from the final report are worth noting because they 
set the stage for the introduction of two Stan;iards which were indeed 
written, not revised, by the General Assembly. Continuing with its focus 
on outputs the Joint Subcornnittee concluded its introduction with a set 
of premises, the first two of which are as follows: 
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1. The basic purpose of the Standards of Quality is 
to establish minimiJfn elementary and secondary edu- 
cational goals that are to be met for each child' 
(to the extent practicable) throughout the Coinnon- 
wealth. ^ '****<:^ 

2. Standards established by the General Assembly' 

.should be oriented primarily towards products - 
(objectives, outputs and goals) rather than pro- 
• ^ cesses (inputs and means), thereby creating a * 
structure and environment for qua! ity education. 

To the best of my knowledge ,this is the first use of the word mini- 
mum in connection with the outcomes of public instruction in Virginia 
and provides the basis for a wide rangijig set of changes in the orienta- 
tion of public education. With these premises in mind the General 
Assembly wrote the following standards: 

Standard lA. THe General Assembly conclude that one of. the funda- 
mental goals of public education must be to enable each student to 
Achieve, to the best of his or her ability, certain basic skills. 
Each school division shall, therefore, give ^he highest priority . 
in its instructional j)rogram to developing the reading, cofrmuffica- 
tions and mathematics'^kil Is of all students, with concentrated 
effort in.* .grades one through. . .six. Remedial work shall begin 
for low achieving students upon Identification of their needs. 

Standard IB. -By September 197S, the Board of Education, in cooper- 
ation with the local, school ►divisions, shall establish specific 
Statewide minimum educational object^es in reading, communications 
and mathematics skills that should t)f achieved during the primary 
grades and diring*the intermediate grades. 

And how was the General Assembly or the public to know if concentration 
was being focused on these basic learning skills and if individuals were 
to be receiving appropriate remedial work? By meaws O"^ tests:' 

. Standard 7A. By September 1978, each school division shall primar- 
ily utilize testing programs that will provide the individual 
teacher with information to help in assessing the educational needs 
of individual students. 

standard 78* Beginning in September 1978, each school division shall 
annually administer uniform Statewide tests developed by the Depart- 
ment of Education to measure the, extent to which each student in 
that division has prpgressed during thejast year in achieving^the 
specific educational objectives that have been established under 
Standard IB. 

— 

Standard 1 codifies a back-to-basits movement for Virginia. Standard 
7 provides perhaps the most ambitious, comprehensive program of diagnostic 
testing in history. If, that is, the standards are to be taken at face 
value. And while certainly these standar*ds, conceived entirely within 
the General Assembly are not to be taken lightly--the **objectives and com- 
mensurate testing prograp are at present in place— there is good reason 
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to believe that the legislature was not fully arware of the implications 
of what ib was doing. ' The cortenittee had been advised primarily by one 
legislative aide, untrained in psychology'or education. While that aide 
read a great deal of background resea'rch^i and whi]e_the committee as a 
whole learned a great deal about testing, ^the final report pf the Joint 
Subconinittee is a melange of the Zeitgeist , theory, errors and naivete' 
As paper after paper was^elivered by the legislative aide to the comn'it- 
tee,. "We began," in the words of one conmittee member,' "to contlinde that 
he knew what^he was ^^Ikirtg abliut when in fact wha,t was being created 
was legislation by inundation— we were simply inundated with j^apers about 
<)bjectives and criterion-referenced tests and so forth." 

The aide had indeed "discovered" criterion-referenced testing and 
proposed it as the oirly reasonable alternative to .current testing prac- 
tices. The following quote from the final report of the Joint Subcom- 
mittee is revealing, of the errors and naivete' al luded to above: 

Far too much emphasis in testing has been placed on, how a group 
of students (a classroom, school, division, or state) compares 
relative to a "norm" group. Relative, rarfkings bear no direct 
relationship to an absolute level of academic competency. 

Particularly with basic skills, knowledge is more absolute than 
relative. Thus, use of relative rankings or percentile scores 
masks any change in the absolute ^acquisition of skills or knowl- 
edge. The Educational Tes'ting Service, which administers the 
College Entrance Examtrviation has noted ^a steady decline over 
the last ten years in trie absolute academic achievement of stu- 
dents taking its examination* "Norm referenced" tests do not 
show the decline that has actually taken place. 

Such delightful confusion could be accepted if it did not accompany 
a document proposing a program that no one had yet accompl ished— getting 
all or most teachers to use tests, and tests, as meaningful, not to say 
diagnostic instruments! ' ^ 

. 'The Department of Education's involvement in this policy change was 
nearly nil.^ Indeed, the Department had been operating independently on 
its own initiative. The Director of Program Evaluation had convened in 
1975 a State Testing Committee made^-up of LEA, SEA and IHE personnel to 
propose* a comprehensive testing program ^or the state. Much, though not 
all of their work was rendered moot by the actions of the legislature 
(the State Testing Committee did no-t make its final report *until December 
1976, some nine months after the action of the legislature),. The Depart- 
ment apparently had no inkling of what the legislature was about to pro- 
pose; the legislature, for its part, did not acknowledge the existence 
of the State Testing Cormilttee, 

V 

The effect of^he new SOQ for 1976-78 was to strength^an equation 
that had been in theNqaking for the preceding two years. That equation 
was simply evaluation -testing. 
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The new SOQ were passed in March of 1976, the Director of Program 
'Evaluation resigned in August and the position remained vacant for nine 
.months. When it was advertised aga^in, the job description had changed 
..coii^iderably. Now. called the DirectoV of Program Evaluation and Testing, 
' -t^ new ad specified a person to provide: 

Leadership in developing a program for measuring individual 
^ student a-ttainment of basic skill-S', developing a program 
for assessing student achievement; coordinating the work of 
staff members 'in developing, administering, and interpreting 
the statewide testing program; and developing and administering 
the testing. program budgat. 

Evaluation = school outcomes = eqyal test scores. A rather differ-- 
t ent orientation than in 1(973. 

It is quite possible that all testing except t'hst* prescribed by the 
new SOQ would have gone by the Boards in 1976 had not Virginia's inter- 
-mittent policy making body^ the press, jumped into the fray. In both 
articles and editorials, newspapers, particularSy those in Richmond, 
the State Capital, argued that the elimination of norm-referenced tests 
would lead to chaos as it had In other states, California was cited as 
a state which had changed tests so often that no one knew where the 
state was, what the anchor for scores was. The NRTs were kept. 

^ A Case Study . 

THE EVOLUTION OF VIRGINIA'S GRADUATION COMPETENCY TESTING PROGRAM 

If the history of the Standards of Quality provides a general framework 
^for the evolution of the equation testing = evaluation, the history of 
"Virginia's movements in the area af "minimum competency testing'' provides 
a concrete example'of that equation in action as well as how policy deci- 
sions are made and how programm&tic decisions ^et elevated to policy levels,. 

In a^period of two ^ and a hajf years, Virginia moved from having no 

competency requi rements' at all, through a stage of having an "Oregon plan" 

with localities having latitude, to having a uniform statewide graduation 
competency testing program. 

In 1976, acting to head off a legtslative mandate and in reaction to 
""the handwriting on the wall", the State'^Department through the State 
Board proposed that ^a set of competency requirements be added to the 
standards for accrediting secondary schools. While' some evaluative infor- , 
- mation was considered— test score declines as reported by 5TS and NAEP 
and fluctuations in the results of the St?te Assessment Program--there is 
no evidence that the decision to add competency requirements was signifi- 
cantly influenced by these results. 
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The requirements dealt with computation, confmunication , social 
studies and ability to successfully pursue post-secondary experiences 
either in the marketplace or in higher education. The first three were 
adopted almost verbatim from a publication of 'NASSP entitled This We 
Bel ieve . The fourth area was derived from the goals of public education 
as stated in the Standards ef Quality. 

In considering to whom this new requirement should apply first, the 
Department and the Soard were guided largely by an intuitive sense of 
fair play. The Department and the Board felt that the children to be 
affected should have ample warning before hitting the barrier. ,In teniis^ 
of the course of schooling, it made sense that the requirement could 
justly be imposed on those children who were about to enter the ninth 
grade. They would know about the requirement ahead of any high school 
years and hav% the full four years to meet the requirement. 

The plan adopted was referred to as the "Oregon plan" although it 
differed significantly from tfie Oregon program in one respect: While 
Oregon allowed local divisions to establish the competency areas the 
'State of Virginia specified them. However, as in Oregon, the determina- 
tion of how to assess competervcy and how much of any competency consti- 
tuted enough was left up to the localities. 

The localities, with little input into this requirement, gave the 
actjon jiiixed reviews and treated it with mixed reactions. Some divisions 
^already had a similar requirement and essentially igncrred the Board's 
* action.* Other divisions took the new requirement Seriously and began to 
plan in various ways to meet it. ^The most common form of planning was 
-to develop or purchase a test. 

Almost immediately, a large number of local divisions began to lobby 
the Board, the Department and varipus legislators to move the competency 
assessment program to a statewide level. Some lobbying efforts derived 
from legitimate concerns, some did not. Many d-ivisions in Virginia are 
small and lack the staff or money to develop or even purchase assessment 
devices in four areas. Others, it appeared, simply did not want the 
bother of the development process and some, seeing a fertile lode of 
future litigation, preferred that the state and not they be hauled into 
court. The pressures for statewide assessment, of the competency areas 
grew during the fall of 1977 while legislative committees were meeting. 

When the legislature met in early 1978, an addition was proposed 
to tfte testing and measurement standard created by the legislature in 
1976. As originally written, the 1978 standard would have required testi 
to be used both for graduation and for promotion from grade to grade. 
This standard was debated hotly over a period of two months, but no mem- 
ber of the Research, Evaluation and Testing staff was called on to provide 
any kind of testimony about the wisdom of the standard- And, unljke the 



' *There some anecdo^fea4---ayjdeciee that some people did not rea-lly 

take the requirement "seriously" until after the first results of the, 

later statewide tests were released in early 1979. 
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original testing and measurement standard which had been preceded by a 
year-long, if somewhat flawed study, no such study was conducted prior 
to the introduction of this new standard. 

The addition- of a standard calling for a statewide' graduation test 
was really no surprise as the local superintendents had gon.e on record 
as favoring it. Calling for the use pf tests to determine promotion, 
however was something of a shock. On what evaluative information was 
this stJ^ute proposed? One cannot be certain because of the lao-k of 
contact bfetween the Testing Staff and the legislature, but it is reason- 
able to Assume that the legislature was responding to what had become 
known as the "Greensville Experiment". To understand the Greensville 
Experiment and its power requires a digression, but it is an illuminative o 

The original Standards of Quality^had called for a division's 
achievement level to'^atch its ability level, 'both levels being deter:mined 
by NRTs*. While most divisions had close correspondences between the 
two tests, Greensville County did not--the achievement level was well 
below the "ability". Based on these results, the State Board singled out 
for public censure the County of Greensville as failing to meet the 
Board's Standards, Greensvnie'y-response was to retain about 57 percent 
of the fourth graders and high percentages in grades 1, 2, and 3. All 
those who did not achieve at a certain level on the NRT were retained. 
While this caused a short term furor within the county, the superinten- 
dent was able to build public support for ther program which involved more 
than simple retention based on test scores. Greensville's superintendent 
presented the program as offering low achieving children "more time to 
learn" and was able to convince a substantial portion of the population 
that it had this benign intent, theret)y removing, much of the usual stigma 
associated with "flunking". The superintendent was able to convince the 
majority of Tiis constituents that it would be unfair to promote the chil- 
dren who had not achieved a certain level on the tests; that it would in 
fact be more humane ta allow them more time to learn these skills by 
repeating the grade than to go on and encounter even more difficult sub-*^ 
ject matters in higher grades. By 1977 favorable reports on Greensville/ 
had been written in various^^tate newspapers. Time magazine, and shown 
in prime time on two of the three commercial television netv/orks.^ 
Greensville's program is complex and difficult to evaluate partly be- 
cause of inadequate baseline data and partly because the need for adminis- 
trative decisions which outstrip researchers' ability to gather data, 
Greensville is a small division with essentially a three-person central 
office. One cah -scarcely fault them for not being research oriented, 
for op.erating with an administrative style of deciding what needs to be 
done on the basis of the collective wisdom^of the office and the school 
board and doing it. However, the general public and some legislators had 
gotten a simplistic notion of the program that said essentially, if you 
fail kids, scores go up. The fact that those children with low scores 



*In fact, as phrased, the standard did not have a^ny meaning. The 
interpretation given to the standard was that the average percentile rank 
for 'a division in achievement should be equal to or above its average 
percentile rank^or* ability. 
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were being retained ^and that this selectivity alone would m^e sgores 
appear to increase was not a part of this simplistic conception. Part, 
but only part, of^what Greensville County ,had done was to test third 
graders locally and retain those scoring below. a certain level. Thus- 
when the succeeding fourth graders took the state required test at that 
grade,. the fourth grade scores appeared to rise dramatically. Other 
increases occurred in other grades. 

By the School year of 1977, however, the children retained in the 
lower grades for one or more years were now 14 years old. They were also 
in many cases, in ungraded classes. However, for purposes of funding by 
the state, the children had to be declared as either eighth graders or 
special education students. By declaring them as eighth graders they 
would be eligible for intennural athletics and they were so declared. 
But being now eighth graders, they came under the law that required all 
eighth graders to take the state NRT. The resulfwas that the scores 
that were apparently high, took an equally apparent nose dive. A legis- 
lative resolution commending the superintendent of Greensville for his 
efforts was tabled and the promotion-byrtest section of the proposed new 
_s.taD.dldrd was. deleteiL .^Such-i-S-ihe^relationshi^) betweerv^^est. data and - - - 
policy^ The standard calling for a graduation test was retained, however, 
and in its final form read as follows: 

It is the policy of the Commonwealth that the awarding of 
a high^school diploma shall be based upon achievement. In 
order to receive a h.igh school diploma from an accredited 
secondary school after January 1, 1981, students shall earn 
the number of units of credit prescribed by the Board of 
Education and attain minimum competencies prescribed by the 
Board of Education. Attainment of such competencies shall 
bee demonstrated by means of a test prescribed by 'the Board 
' of Education. 

Certain characteristics of this policy statement and its concomitant 
action r^qu;jrement stand out. The policy is, in one sense extremely prar 
scriptive in that a test is required. In another sense, the policyi is 
extremely liberal in that no areas are defined for the testing. ^> Unber 
the letter of the law, the State Board is free to prescribe a test; in 
the basic postures of hatha yoga. In one sense the policy is .extremely 
vague in that it does not define test. And a number of people were con- 
cerned that bedause the standard referred to "a test" the Bjtefrd was not 
free to pr^escribe different test<f in more than one area. - 

Needless to say, hathaXyoga is not a requirement for graduation and 
test has been Interpreted as four-choice multiple choice. The Board 
decided that they could require more than one test—reading and mathe- 
matics—and no legislator has complained that this violates either the 
letter or the^ spirit of the standard. That such concerns about wording 
could be raised and discussed with a semblance of seriousness is indi- 
cative of the sometimes f ragile^elationship among policy making bodies. 
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^hile the legislature altered the policy of how competency was to be 
demonUrated, it left the year when the statute became effective unchanged. 
The class of 1981 was (and is) under the g^n. Again, no one from th% 
Testing Staff of the Department was asked to testify concerning the effect 
of these policy and action changes. At this point, no litigation had been 
resolved in Florida and McClung had only recently {197S) suggested that 
the length of phase-in time* for such a program could be a source of liti- 
gation. The Department 'felt strongly, however, that the children to be 
affected should be tested as early as possible. Early testing would allow 
for remediation and to prevent any feeling that the rules of the game * 
for'getting a diploma had undergone a sudden andtcapricious change in the 
eleventh hour (or the eleventh grade). Accordingly, the Department convened 
two committees of reading and mathematics experts for the purpose of review- 
ing tests in these areas and' recommending to the Board which should be used. 

It should be noted in passing that the catling of these committees 
reflected a long standing Department policy. Whenever changes. in educa- 
tional programs are likely to have 'significant impact on local school 
divisions, conmittees representing different region^ or different levels 
of organisational stqitus or both will likely be convened to "evaluate" 
the change and, advise the Department and Boards In my experience, such 
commjt tees function substantively, not as window dressing. 

The two cofrmittees consisted of teachers, supervisory staff, univer- 
sity faculty and representatives of various special interest organizations. 
Their charge was to review all tests that hai^ been developed in Virginia 
under the "Oregon plan" years, as well as those tests that were coming 
onto the market from conmercial publishers. Two tests were redormiended to 
the Board and accepted by it in June, '1978.. 

With the tests chosen, another question arose: How much^ competency 
, on these tests is enough? Where Is the cut-score? This issue might seem 
at first to be too minor to constitute policy. On the other hand, it 



:ertainly meets all of the £xite^ia for a, policy issues as described by 



Mann and cited in this introduction. It is a public problem, with impor- 
tant consequences/; with political, economic and moral dimensions, involv- 
ing a goodly amount of uncertainty and viewed very differently by different 
interests. In any case, the amount of heat generated by the debate elevated 
the decision to the policy level. * 

Most of us directly concerned with testing were largely in agreement 
with Gene Glass (1977) that cut score decisions, cogld not be based solely 
on'technical considerations. We werep, likewise in agreement with Glass 
that: ' ' . * . 

For most skills and performances one can reasonably imagine • ^ 
a continuum from "absence of skill" to "conspiciuous excellence. 
But it does not follow from the ability. to recognize, absence 
of the skill that one can recognize the highest 4^vel of skill 
below which the person will not be able to^ucceed (in life; 
at the next level of schooling, ofr.in hi s/^chosen^ trade) .. . If 
anyone would dare^ to specify the highest level of performance 
below which no person could succeed 1n life as 'a parent, 
counter examples of persons reading below the minimal level 
' yet who are regarded as successful parents could be supplied 
in abundance. 




6r\ the otheY hand, a cut score was necessary and thus the sear^cfv 

had tD" be confined ^to methods that would reduce arbitrariness in its 
bad sense of being capricious. Several methods were considered and 
that which seeriled most congenial to us was the method proposed by 7 
Jaeger (1978). While openly judgmental , the procedure allows for judg- 
ments by various audiences and a series of iterations before reaching' 
a final decision whiclj allows j^es to ^ange their minds^'based on 
new information. However, one of the cycles of the model requires 
that data from item field testfs be provided to judges that they may 
nothowjnany cJhildren Xctually answered a given i.tem correctly. This 
cycle seems particularly important 'in view of other similar models 
which operate without actual data and often lead to unreal istically high 
scores (e.g. procedures for valida^ting the National Teacher's Examina- 
tions). ^ 

Hpwever, with the legislation enacted^ in March and testing scheduled 
'for November, clearly, no field te^t data was going to be forthcoming 
and the two tests chosen had limited field testing and noae in Virginia. 
In June of 1978., the testing staff recormiended that the setting of the 
jflut score be delayed 4ntil actual test results were back and that' the 
result^ from the firs-^^ministra-tion be used in the Jaeger model where 
data was. called for, jSk 

No action one way g^r another was taken on this recommendation for 
some months'. In late^ August or early September the press became ^cogni - 
zant that no cu^core would be estabjished until results were in, and 
in several cities allo^d as how such a procedure would permit the scores 
to be ''tampered with'* to make the resyits both polemically and econom- 
ica-llyacceptable to the "educationist establishment". In reaction to 
these allegations in the press, tfre State Board attempted at its Septem- 
ber meeting 1?S set a cut score in the absence of arv^ procedures. "Le-t's 
set a score and let^e chips-fall where they may, said the President 
of the Board. "We Wn always change the score Jf we need to"! I and 
two other mei^erS'Of the department argued vehtknently against 
this approach, and after several heated exchanges, won a month's delay 
to conduct some kind of study only by pi^omi^r^ to provide a recommended' 
score at the next Board mea^ting. During tmFmonth ,- seven groups of about 
15 people each were convened in various diyi^fons of y;he state at the 
invitation of .the local stjperintendent^of^ the division. The instructions 
to thq superintendents were to' pick people representing professional 
educators" of all levfelsf^ ahd parents and other interested community mem- 
bers. Each item was^examined in terms of its importance and then a glo- 
bal rating was obtained as to what would be a fair passi<iq score. The 
range, of scores' was ,35^to 85r When analyzed according to a lay-^ucator 
omy, Uy persons wanted a score around 75, educators aiAiund 70. 
Departmerft recommended 70 xh^ch was accepted by the Board. 

It is worth* noting tWt feelings among the testing staff were so 
in opposition to this ^haphazard approach that others ij\ the Depart- " 
ment administration "absjmed" us from any responsibility in the con- 
duct of the modified, Tess-than-rigorous cut score study. -Jji retrospect^ 
I'feel that this cut score was predetermined, that it was not likely to 
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, have been anything other than 70 (the judgment pf the groups actually 
favored a score of 75 and had the testing staff been in charge of the 
. ■ «recommendatfon, that Would no doubt hav,e been the recommended score; 

the testing staff had conducted the study it is not clear where the - 
cut sdbre would -have fallen). 
# • t . • , 

A^ain, in retrospect, it 'was good that 'the cut score^as in fact 
, established before the test was. given. Even though int)ie actual pro- 
cedure political considerations and public relations con's i derations 
weighed more heavily than conceptual soundness,- a technically sound 
study conducted before the test administration might have led to more . 
- ' . "<isj^jjption and di-slocation" in Glass' phrase and a cut score set after 
thej^st administration certainly- would have miximized disruption and 
dtslocation. This unfortunate outcome mjdid have been determined by 
events surrounding the test administration that could not have been 

foreseen prior to that administration. ^ = " 

• • . Jt) 

The competency testing program had been good copy for the media. 
Both print and electronic media representatives had been keeping close 
tabs on events surrounding the competency program. When ^he test was 
„ ^ actually administered, schools were deluged with reporters interviewing 

# 'Students for their Veactions. The tests, said the students without 
% exception, were "a piece of cake" an4 "anjnsult po my intelligence", 
etc.^ Apparently'no reporter questioned whether or not his sample might 
*be biased— that children who experienced difficulty would Aither steer 
cle^r or the-cameras or be loath to admit hardship in the presence of 
.peers sneering at the ease of the test. In' at least one instance we 
obtained reports that only the "right" students were being aimed in the' 
direction of the camera^ by the school staff. 

In Addition, approximatel^a month^after the test administration 
and before any results jwere received, sample items of both tests were 
released at a news conference. These items did little to convince the 
- press that the test >iad been difficult and one paper published the items 
with the first of a series pf vitriolic editorials attacking the test 
as ii farce and public education as a sham, bilking the public by permi-t- 
ting children to graduate knowing so little. 

^One can only ii^igine wha^ would have happened in this charged atmos-. 
pte^re jf the cut-score' had been set after all this publicity. It is 
likely that any group charged with establ ishijig a passing ^ore would 
ha^e felt obligated to set the score higher than what had been established 
prior to the administration. With the score set at 70, 9% of all white 
students and 33% of all black students fa*ed the reading test. .If the 
score^had been, say, 80, these figurea would have risen to 25% and 63%, 
respectively. 

Before leaving this particular topic, it should be noted that while 
the data preferred by the testing staff could not be gathered, some 
infohnation was gathered, for use in establishing a cut-score. Some 100 
people did render judgments about the appropriate cut-score. The extent 
to which this data influenced the cuf-score of wfiich it "improved the ^ 
decision making process" is undeteminable. Khat is clear is that this 
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is the type of information \hdX will often have to be used in informing 
policy makers. Because of ttime pressures ,^p^litical , economic and publjc 
relations considerations, itys likely that the need for a decision will 
usually outstrip the professionally desirable methods for*cdMecting 
information. If evaluators are. not simply going to take their marbles 
and go home (thereby rendering their contributions nil), they must learn ' 
to cope with ^nd use less than "pure" data. -The techniques for such 
coping and use and the criteria for evaluating the power of such data 
are by no means clear , althougti the exponential increase i« evaluation 
methodologies--everything now seems to be a metaphor for evalu^ion— is^., 
testimony that evaluators are alt least aware of the -problems . 

With a cut-score set, the tests given, the question now arose, as to 
how to release the data. For many years test _scores were not rel eased, 
but in 1971, the State Board, reacting to both public pressures for such 
scores on a di vision-by-divisiorT basis*and an opinion of' the Attorney 
General, decided to make such release a matter of course. In s^ite of 
an attempt to defuse invidious comparisons among the divisions'^y empha- 
sizing the desired match between "ability" and "achievement" (noted on 
page 16), this ^practice did not sit well with many legislators as noted 
in the statements from the Joint Subcommittee report. Nevertheless, the 
Board's decision was not ^h§llenged and the Department planne^^a release 
in this fashion for the Gradua^tion Tests. In addition, because of results 
already reported in Flor^ida and North Carolina" showing marked racial 
differences, it was decided to report the data analyzed by these two / 
categov^ies. The results were accompanied by a brief summary-interpreta'- 
tion- prepared by the testing staff. ^ ^ 

The racial differences >n passing rates made bj^headl ines, around 
the state. In addition, a'^ number of divisions that do not traditionally ' 
do well on the norm-referenced tests did ver^ well on the Graduation 
Test, much better than neighboring divisions with similar NRT scores--. 
This'led in some instances to'phone calls to the Department questioning 
the validity. of the results for- the high scoring divisions and in a few 
cases to open charges that some divisions had somehow cheated--had taught 
to the'test, withheld certain percentages of their kids who should have 
b|en test^, etc. ^ ^ % 

5uch accusations place the Department' in a difficult position. It 
Tacks the resources to administer the tests and certainj7 cannot verify 
the qhallefiges of impropriety.' The best that the Department can xjo is 
to prepare a narrative to defuse these kinds of attributions. However, 
this course has its owrt dangers. If the narrative is too long and makes 
too many points, the^ media accuse the Department of managing the data 
and ignore the narrative. 

If the harrative contains information which the Department feels is 
important but which i^ complex —which cannot be dealt within a few 
paragraphs or during a 60-second segment on the evening news— such a, . 
narratfve is also likely to be ignored. For Example, while the failure 
rate for»J)lacks was four times that of whites statewide, this rate did 
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not prevail in all (jivisidns. Indeed^ in a few di vi sions ,'\blacks passed 
"at a higher r'ate^than whites, and an examination of those d>v4s^ons 
where blacks performetl well indicated that the results could not ^be a 
simple function of demographic variables,' SES or anything simple. In all 
•lixeH^pood they reflect subtle program and extra-school variables. But 
sublety has no place in^the face qf deadlines and* short on-camera reports, 
although the variations 'di scussed above were reported In the narrative. 

The point of this disrcussion >s that evaJuators in possession of 
"public inforn^tion'' often have a difficult time getting that informa- # 
^ipn to the public in usable form. The Dei#rtment is often forced to 
do a "data dump" and hope for the best. Presentation of what the Depart- 
ment considers important is often viewed with ^suspicion as being t^iat 
v/hich tjie Depc^rtment considers to be in its best interest. 

I wokfld note in conclusion, however ^ that in fact there is never 
really a* "dataj dump" , but onl^^ a method and format of reporting certain 
information to^ which the press and others have become accustomed. It 
has been noted. by numerTOs ohilosophers of science that "data" should 
really be called "capta"-^that , nothing is given but rather is, taken, 
captured.. The problem for eyaluators is to obtain acceptance for the 
kinds of capta they consider relevant , 'not simply those whjch may be 
reJ event to the press. * . : 

• ^ POSSIBLE^Fp^ru FOR EVALUATIVS INFORMATION 

' ) In t>ne preceding sections of this paper, we have seen that the 
influence of evaluation on polic^in Virginia has been a very limited 
one, having been constrained by the stnjctuical context, as well as 
historical events. The structural constraints were two-fold: an absence 
of evaluation staff a/id strictures on the, flow of information which 
would make it ^difficult to get relevant informatipn to those making 
.policy decisions. The Zeitgeist has also constrained the influence of 
evaluation in two ways. People have not §een evaluation as particularly 
relevant or important to policy except in a negative way. S^condlyr 
the spirit of ^the times says that in fact, the evaluation data are already 
in: * Test scores are declining, costs are going up, teachers cannot 
teach and the'^whole structure needs revamping. This aspect of the 
^ Zeitgeist clearly includes the' general public, as can be seen in the 
annual Gallup polls on pufelic opinion of education, as well" as the 
policy makers .as can be seen from the number of legislated educational 
programs, * * - ' * " 

In Virgir3,ia, one senses, though it ts difficult to demonstarate 
with "hard" evidence, a punitive, calvinistic attitude towards- the 
schoaH. The schools have failed to control' the natural depravation of 
^mankind and mu*s^ be punithed, for their .shortening. 'It*is never phrased 
that way, of course, but the behavior speaks louder than the words. I 
have often been asked by people outside of Virginia what remedial pro- 
grams the state is {fcpviding for those who fail the Graduation Competency^ 
Test. My answer is lione, there is no money for remedial programs. 



When the same enquirers have asked "Is this not a little unfair"', my reply 
is this: Many^ho could .provide funding believe that if the schools/ were 
doing their job there wouldn't be anyone failing the test and so nd addi- 
tional morfey vill be forthcoming to assist what should be done anyway. 
Not all policy makers feel this way but enough do to prevent any sums be- 
ing^appropriated for such programs or for; legislation being introduced to 
provide money,* 

Similarly, the Standards of Quality can be assessed in tims gf their 
per-pupil cost. That is, how much money on the average/ does it cost to ^ 
provide the programs required by the Standards of Quality, According to 
data collected at the local level and st;^ate level,, the standards have cofi- 
sist^ntly required more money than the legislature has appropriated for 
them. This failure to , appropriate actual costs for constitutionally required 
. standards, largely written by the 1 egislatureg^fl ects again a certain 
attitude of punitiveness towards public educa^mn, 
% 

As another instance of constraints or> the ase of evaluation at the 
state level in Virginia, let 'us consider a situation in no way unique to 
/VicS^hU4i but rather symptomatic of, certain organisational structi>res, \ 
Evaluation, to be used properly , "must take place in'an atmosphere where 
there is some freedom^tp fail.' If one is not doing pseudo-evaluations 
where the results are "det^rm^ihed beforehand / one cannot guaT^ntee the out-^' 
come of an evaluation^ Such freedom does riot exist in education and such 
ana such freedom does not usually exist in bureaucratic structures compet- 
ing with other such structure's for money and power. Rich (1979) distin- 
guishes two different ways of "avoiding risk" in organizations. For 
scientifically oriented academic researchers , reduction of risk consists 
of and in the fdeal requires, new information which may contradict earlier 
information. The goal is truth and ^he new iQformation reduces the risk 
of being wrong. While I think Rich's description of academic's ooen arms 
acceptance of new information is self-serving and overstated'^, certainly 
ther§ is more freedom to fail in academic settings. 



*After the first draft of this paper was written a part of the 1980- 
82 Standards of Quality was changed to permit the hiring of extra personnel 
for eighth-gwders scoring three or more years below grade level on the 
eighth-grade norm-referenced tests given annually. These children are 
kno^j^n to be at some ri^^ in .terms of passing the Graduation Competency Test. 
The cliange thus has the effect of providing remedial assistance although 
it is ^not' phrased that way. This kwid of incflrect use of evaluation informa- 
tion IS becoming more common in Virginia. . * 

^he case of acupuncture vs. the Anerican Medical Establishment comes 
Vmediately to mind. Rich's statement miaht be*better phrased, to the 
effect that new Information is welcome inrdirect proportion to its poteri- . 
tial for getting grants and publications for the receiver of the information. 
Resistance to change in the scientific community has been beautifully and 
amply documented in Thomas Kuhn's The Structure of Scientifl^Revolutions . 

/ 

so ^ . • ' 



Rich is^more on target yrfien he speaks of the Manager's Perspective 
of Risk Avoidance. Here risk relates to competition for scarce resources. 
The manager in a bureaucracy is likely to ask how new information might be 
used to embarass him or to place another agency in a more favorable light. 
Evaluation iti this context does not have a "truth" orientation, but a 
political one. Evaluation will be well received if it helps the organiza- 
tion and manager meet goals, minimize costs and maximize gain. This means 
that evaluation as customarily defined has an uphill fight--it takes place 
in a value-laden political context. Education is in an area of competi- 
tion for scarce resources and in an era of negative opinion towards its 
achievements. In such an area in such an era the cost of looking bad is 
too high to permit a scientifically detached perspective^ on evaluation. 

Finally, the role of evaluation in Virginia is co/^trained because 
it has been narrowly defined, namely by test scores. ^It is safe to say 
that until test scores cease to be an area of concern,^ such potentially 
fruitful sources of evaluation as affective variables or process vari- 
ables will be ignored. Similarly, unitl test ;cor*es per se become a non- 
issue, little will be done»in the way of process or formajtive evaluation. 

It would be easy after reading-the above paragraphs to conclude that 
evaluation is unlikely to ever be usefully applied in Virginia through 
the Oepactjpeat of Ei^ufatfon^ Easy » -but wrong, i^hile ^eats dp not^auguc^, 
well, there are areas where i'maginative and ene^etic use of dkta- and 
lobbying by those^lnterest&d in evaluation can produce results. 

The present press for accountability education is likely to in- 
crease. The problem for evaluators is to eliminate the equatipn of account- 
'ability with test scores and write a set of equations including man^ 
variables. While there i^ no guarantee that such a conceptual oroadening 
can occur or be used properly, there is at least one area of accountabil- 
ity where it might. The focus on competency in Virginia has moved from 
students ta teachers 'and while there is, yes, a test required for new 
teachers ,^ al 1 concerned Vecognized that such a test will not' guarantee 
anything excej>t the screening'' out of near-illiterates. There is a 
recognition that knowledge and competence are different and that com- 
petence is related to behavior. This is not^a recognition well articu- 
lated ^et, but'it'is there and^the necessity 'to move towards assessing 
teacher performance would seem to 'at least open the door tO' a broader 
cpnceptual ization of evaluation, 

'Similarly, evaluations could, take pol icy tissues and relate them to 
empirical research in such a way as. to broaden perspectives and hopefully 
open the way for -better utilization of information. For example, class- 
size has been a policy issue in Virginia for some years. There has been 
a mandated reduction in class-size as apart of the SOQ for some years. 
The mand^e occurred without data. Now,' Glass and Smith and others have 
provided-some widely accepted research findings relating class-size to 
both achievement (Glass and Smith, 1979) and affjBctiye outcomes (Smith 
and .Glass? 1979), The key, it would seem would be to identify those policy 



areas that already are issues and try and provide as moh evaluation 
evidence as possible even if the evidence cannot' be collected b:^ the 
state department. 

The- job description for my position has undergone considerable 
rev.ision from that given on page 14 and puts emphasis on evaluat^ion. 
as noted earlier, however, this titular emphasis has not been backed up 
with a staff adequate to the job responsibilities. 

Evaluation related information has_ come into prominence in recent 
years— even if it has not always been used properly. A decade ago, no 
one in Virginia was particularly concerned about assessment. Now tests 
are everywhere. A thorough history of how tests use grew in kudzu- 
fashion is beyond the scope of this paper, but such a history might pro- 
vide 5ome clues about how-to build interest both in other means of assess 
ing outcomes and assessing other outcomes. 

-Finally, those interested in evaluation must just keep -pushing evalu 
ation as an important and useful activity. In the two and one half years 
th^t I have been with the- department , workshops have been held both for 
persons in the field and in the department on techniques of evaluating 
projects and proposals, , Each time a new program activity is Dropbsed, 
*the^question is asked (by^mej how are you going to Evaluate it"? Ttie 
e^Ctent^.to wfvfch t'His awarene'ss jDuilding activtty'has been productive is 
not fully known but there are encouraging signs that evaluation is being 
considered more as an integral part of programs from their inception. 
This is particularly true in formative evaluation mere the information 
is used more as a guideline for^ ^Turthejfc^ction than as summative judg- 
ment and hence maximizes the risk avoidance necessary in a bureaucracy. 

In conclusion, it should be noted that "evaluation" is also chang- 
ing. A decidde ago, evaluation was conceived largely in terms of the 
laboratory model of research described by Fox (1977).^ -The inadequacy of 
this- model has been widely recognized and there are now a plethora of 
papers drawing from fields other than psychology and education to pre- 
scribe techniques for educational and psychological evaluation. There 
has been, similarly, a recognition by many that evaluation does occur 
in an environment where power, prestige, and economics often have a 
higher value than "scientific rationality". The degree to which changes 
in evaluation will produce models more relevant to more audiences is not 
clear. The degree to which evaluators will, be willing to partici^pate 
in "impure" rese.arch, the -point at which they will f^el that their hands 
are too politically dirty is also unclear. One must^hope that the 
activities of evaluators will produce, a better match bei:ween policy and 
results than now exists, and work to. that end. 
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Michigan Department of Education 

■ ■ / 

^ HISTORICAL BACKGROUND 

The State Board of Education 

The last f if teen. y^ars have been a period of transition in 
Michigan education. Historically the governance of education was 
delegated to local boards of education. The State Constitution, 
statute§ and regulations tended to provide wide parameters for local 
programs and policy waking. The Department ^f Education initiatea 
little policy and was often criticized as a ''do nothing"' agency. 

The early 1960s set the sta"ge for change. Prior to 1964 the State 
Board of Education Consisted of three members, plus the Superintendent 
of Public Insti^uction. The memoers, including the Superintendent, 
were elected at a biennial spring election. Since voter turnout at 
the spring election was usually small, educa^r organizations found it 
easy to influeace th,e election of persons friendly to the concept of 
local control of edufcation. The constitutional authority of t^e 
Superintendent was stated as "...shall have general supervision of 
public instruction in th^ state..." and, "...dqties and compensation 
^hall oe prescribed fyy law."l The explicit authority of the Board 
was, "...shall have gei^eral supervision of the state normal college - 
and state normal scbooJLSf and the duties shall be ^^pr escribed by 
l3w.*^ Tbe State Superintendent and Board gave direction to a 
relatively small Department of about one hundred thirty professional 
and clerical employees.^ The authority was weaK and the resoUrtes ^ 
needec to govern a state educational 'system of over 700 school 
^districts providing instruction to nearly two million students were 
jnadeqajate. Thus, few policy initiatives emanated .from the state to ' 
9'ive direction to Michigan education. 

The mid-sixties brought together several changes in thought, and 
several events in Michigan and the nation, to produce a different 
State Board and Department. The base change was incorporated in a hew 
State Constitution 'aaopted in 1964. It redefined the role of the 
State Board of Education, and changed the election process. 

The 1964 Constitution established an eight-member bCard with 
candidates nominated at party conventions and elected at the regular 
fall biennial elections. Terms of office 'were made eight yeaifs. The 
membership of the Board was completely changed. A more subtle change 
than the election process and membership was the attitudes of the new 
members. They tended to have* constituencies beyond education, to have 
social concerns beyond education, and to have political ambitions 
beyond the' State Board of Education. 

The new Constitution expanded the role and authority oi the 'State 

Board of Education. .The State Board dujties were to be, "leadership 

and general supervision over all public education, including adult 

education and instructional programs in state institutions, except as 

to institotions of higheij education granting baccalaureate degrees, is 

vested in a state board of education. It shall serve as the general 

planning and coordinating body for alL gublic education, including 

. • ^ ^ -x 
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higher' education, and shall advise the legislature as to the financial 
::equirements in connection therewj^th • "-^ The persuasion of the Board 
was to develop a' program of overall supervision of education and to 
initiate*'^policies in keeping with their leadership mandate. The State 
Board was the policy board for the Department of Education. "The State 
Superintendent was both the Chairpetson of the Board and the Chief 
Administrative Officer of the Department. The Department .of the past 
had ^to be changed to be responsive to the active role the Board wanted, 
and to' the n^w world 'of education. 

The Department of Education 

The events of the time provided a setting. A collective bargain- 
ing statute was enapted, aijjd old power structures were being altered. 
Th^. courts increasingly entered into educational matters. Educa- 
tional issues were often being identified and defined outside 'the 
educational community; there was a netfd for strong leadership in 
Michigan education. 

The passage of the Elementary and Secondary Education Act in 1965 
(ESEA) not only increased the Federal presence in education, it in- 
creased the role of state agencies in education. Most of the ESEA 
pro^?;ams' ^-flowed program money through the state and provideti adminis- 
trative funds to the state. The resources available to the State 
Board and Superintendent were increasing and with it their ability to 
in-itiate policy was increasing. 

As State Board members began to address their new responsibili- 
ties an obvious question was posed, how "healthy" is Michigan educa- 
tion?- Answering the question was more difficult than expe|:ted. Al- 
though the Department collected some financial and staffing data and 
issued a few statistical reports, virtually no evaluation activities 
existed. No effort was made to gather and analyze a broad range of 
information about the scho^s and districts of the state. Certainly 
there was no effort to identify inadequacies, inefficiencies and 
inequities in the system. This paucity of information presented the 
policy makers a dilemma, a desire to provide the leadership for educa- 
tional Iffipi^ovement, but no base of information about what changes 
were needed most. There were beliefs that some districts supported 
education at a much higher level than others, that learning levels 
were disparate,- and' that the conditions for educating were vastly 
different throughout the state. 

Members of the State Board were anxious to pierform in their 
leadership role- and were not willing to wait until the Department 
could develop an evaluation capabilit;y and produce the information 
they needed. Rather, the State Board, with support from the Legis- 
lature, contracted with Dr. J. Allan ^Thomate of the University of 
Chicago to do a thorough study of Michigan education. His charge was 
to gather together information on the system, to describe the system, 
and to offer recommendations for improvement. The study took eighteen 
months and culminated with a report issued in the fall, '1967. The . 
report drew several conclusions, among them: 
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1. There is a great variation in the educational 
opportunities available to students in the 
S'tate of Michigan,^ and 

2. The Michigan State Department of Education 
should expand and strengthen the Bureau of 
Research, Planning and Development.^ 

The report was well accepted and thoroughly read by those inter- 
ested in Michigan education. The report was a good base from which to 
set a direction. The goals of the 1970s were to be greater equity 
and^ equality in Michigan education, and evaluation, in the broadest 
sense, was to provide the leverage for the changes. 

Initiating State Assessi^ient 

Acting on fehe^rtcommendations of the "Thomas Study/' the State 
Superintendent,' Dr. Ira Polley reorganized and enhanced the, evaluation 
capabilities of the Department by creating the §^reau of Research 
within the Department, Staff for the new Bureau was hired from bright, 
recent Ph.D. graduates of universities like Chicago, flolumbia, Illinois, 
Oregon and Michigan. These persons nat only brought new and different 
skills to the Department, but also the commitment to use these skills 
for educational imprCJvement . Nearly as soon as the small staff of 
four or five were hired, they began to discuss the lack of reliable 
information about the statys and progress of educational achievement 
iri Michiga^ This small group was familiar with, and intrigued by the 
research aiK writings of such menr as Benson, Fox, Hdland, Coleman, 
Thomas, Levin, Bowles and gt^ers.^ The group generally 'embraced the 
input-process-output research model used by many of these investigators 
and saw as important to the state agency the answer to the question: 
"what are the correlates of educational success?"^ Thus, staff dis- 
cussions led to the development of a paper which suggested a statewide 
assessment effort to determine the status and progress of basic skills 
achievement and factors related to it. The paper was shared with the 
State Superintendent who was very positive and asked for alternative 
strategies for implementing the idea. ^ 

> 

As the Superintendent and staff were considering the statewide 
assessment of achievement, the State Board was looking for ways to get 
information on the quality of education in Michigan. They were inter- 
ested in the accreditation function then being fulfilled by the Univer- 
sity of Michigan. 8 At its meeting of January 15, 1969 the Board dis- 
cussed the accreditation process. It was obvious from information 
which came to the Board that taking over accreditation could not be 
easily accomplished. ., the University wanted to keep ifc. More importantly, 
the lack of ^ny demonstrated r,elationship between accteditation and 
achievement was a major concern. Thus, when Polley introduced the 
assessment idea as an alternative, it was well accepted. Staff were 
asked to provide plans for a statewide assessment for the State Board *s 
review. Proposals were placed before ttie Board in January,^ February, ^0 
and Aprll^l and were thoroughly discussed and revised. In April the 
Board passed a resolution which directed the State Superintendent to 



seek legislation .that would provide the authority and funding needed 
to carry out an assessment in 1969-70, and to, do long range planning 
for a more comprehensive program. The Board emphasized ^that the 
basic skills assessment shoul.d also include information' about the con- 
ditions under which the schools operated. 

I 

The legislature during the session l#ad ttiiree other evaluation, 
'assessment, or statewide testilfng bills introduced. In addition, the 
Governor's "Blue Ribbon" panel was about to recommend some kind of 
state assessment as part of the education reform package. While the 
task of getting the authority and fundiug fpr the assessment was not 
easy, the timing , was right for a^proval^; The State Superintendent was 
successful in gaining legislative support, and the assessment program 
was added to the Department appropriation bill for 1969t^^ 

The Governor signed the bill in August, 1969. The mandate was to 
administer a statewide assessment of t;he basic skills prior to January 
31, 1970, Staff began iinmediately '^to plan for an assessment which 
would yield reliable data on reading, English usage, and mathematics 
skill levels for Michigan school districts ,,. provide an indicator of 
the level of basic skills achievement among the districts so that the 
dispairities could b'e described and policies considered to address the 
problem areas. For the first time the st|te agency would be gathering 
information about: the levels of achieve^aent in the school districts of 
the ^tate. The State Board would have information about the system it 
was to super«7ise. 

Implementing State Assessment 

The State Board in cooperation with the Legislature and Governor, 
had taken a major step by obtaining approval for a state assessment 
of the basic skills. However, before the first assessment was done in 
January, 1970, there was> a changing of the guard, Ira Polley resigned 
as State Superintendent and was replaced by John W, Porter. In making 
this choice the Board made a commitment to a pi^o-active and highly 
visable role for evaluation. Porter brought a philosophical commitment 
to use of data in the management of the educational enterprise at both 
the state and local levels. To Porter evaluation was critical to 
managers. He was to define educational evaluation as, "a. process -of 
obtaining, for decision making purposes, inft>rmation concerning educa- 
tional activities, "-^^ and emphasized his commitmet\t by saying, ",,.we^ 
are committed to developing educational evaluation into a fruitful and 
productive exercise. We, in Michigan, are not content . to treat evalua- 
tion as that useless exercise required from on high that takes time 
and pain to produce but which has very little significance for action.'* 
Porter, as State Superintendent during the 1970s, was to be the driving 
force behind state efforts 'in evaluation, and personally used the data 
provided to him. 

Porter tcu?k office in October, 1969, The first state assessment 
w'as conducted the following January, 1970. The 1970 administration 
included the collection of data on student achievement as previously 



noted, but also incruded "data on the socio-economic levels of the school 
and district and general pypil attitude^. This was accomplished by 
administering a "General Information" questionnaire which contained 
twenty-six questions. Students responded to the questions anonymously. 
The purpose of the questionnaire was to provide the information needed 
to estfemate the, group socio-economic status, and the^pupil attitudes 
tj|^'ard ^elf and 'school for each school and district . This was seen as 
necessary information to describe the conditions of education for the 
'State Board, and for comparing group test results from year to year. 

^ , Some groups saw the questions in the '^General Information" part 
of the assessment as nnrelat^ed to the purpose of assessment of the 
basic skirls, and even worse, an invasion of personal privacy because 
of the questions asked as proxies for socio-economic status* The press 
picked up' the complaints of educators and parents, and then legislators 
get into the debate. Department staff spent considerable time and 
effort explaining the need for these data, and defending its cpllection. 
Finally, as time passed and other issues arose, the controversy abated;' 
but was to rearise each year until the State Board in 1973 directed the 
State Superintendent to eliminate the socio-economic status feature.* 
It was recognized that these data were valuable for the proper analysis 
of the basic skills assessment data, but it was just not politically* 
viable to keep this instrument as part of the program. The policy 
decision to eliminaD^* it ^as made on political rather than technical 
grounds. At the same time the SES feature was eliminated, the questions 
used for constructing the attitude scales |^ere eliminated because of 
technical deficiencies. Although these were corrected a year later, 
1974, they were never reintroduced to the program. 

"Mother controversy the first year was raised by legislators at 
the request of the:^ constituents. They attacked one of the reading' 
passages in the test^ because it"was, '*a blatant attempt to inculcate 
anti-American and anti-free ent'§rprise values in school children/^-^^ 
The Department staff discussed these issues with the legislators and 
were able to avoid serious action against the assessment program. The * 
compromise solution included changing the reading passage for the next 
year. 

The time between pa'ssage of the assessment legislation and its 
first administration was only four months. The short timeline did not 
allow the ^development of the long range plan for assessment. Tke lack 
of a plan produced uncertainty and distrust among local educators over 
the^'ultimate purposes and uses of the tests and the data they would 
yiald. This opposition was further stimulated because the program 
was new, it was championed by a new State Superintendent witti whom 
they had li|:tle prior experience, and it was ^n intrusion by the state' 
into local educational autonomy. The State Board ,^nd Department were 
seen as pushing out to exercise new authority and were using the 
assessment program as a vehicle. The program became the focal point 
of opposition. 

The results of the first assessment were sent to local ^school 
districts without fanfare. There was some interest from the press 
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in repoizting "scores" of local districts,, hut the Department deferred 
requests to local districts, or were able' to convince the media people 
not to report them. 

The Department used tlie results for two major purposes, -1) analysis 
of the correlates of educational success, and 2) as ofie o'f the criteria 
to determine school district eligibility for state compensatory educa- 
tion funds. The aaalyses to identify correlates of school success con-,, 
firmed other such studies; 'the coM:^lations between m^an achievement 
as measured by the test and percer|t minority students were about .50^^ 
and. achievement and mean 5ocio-economic status of students were about 
.60.-^ The analyses were disappointing because all other correlations 
between composite achievemeyc,^ and 'expenditures , staff training, 
salaries; size of districF; etc. w6re less than ^20.^^ Again the vBri- 
ables in control of .the school manages were disappointing. The Depart- 
*ment was to repeat the studies the next year with the same results, 
and then dropped such analyses frpa^he program. 

The second 'major state use of the results Vas as one of the criteria 
for determining. school district eligibility /or funds under the state ^ 
compensatory educacion program. Eligijii'lity for funds had Previously 
been determined cxn the basis of socio-ecoq^mig indicators (e.g., similar 
to the ESEA Title I use of aid to dependent children, fi|5ily income, 
etc., as indicators). There was *a strong feeling among Department staff 
'led by John Porter that eligibility should be more directly determined 
by a measure of • "educational deprivation," i.e., low basic skills 
achievement."* k position pap^ wHrch advocated the use of mean disl^Pit 
scores on the state assessment as one of the criteria for eligibility 
was developed, and adopted by the State Board of Education, The idea 
was well_ received in the legislature, and the state compensatory educa- 
tion program. (Chaptei:* 3 of the State School Aid Act) was amended for - « 
1971. The policy makers strongly believed the direct achievement 
measure w$$ 'a better cri^rion for directing funds to alleviate low 
achievement problems. Latef, Giapter 3 was amended to make res\^lts on 
the state assessment the sole criterion for district eligibility.. 
Chapter 3 will be more thoroughly discussed in a later section-^^g^^j^his 
paper. 

State Assessment: Local Educator Reaction ' ' 

The state assessment the first year had* been authorized and funded 
through the Department budget biil. This was an expedient method, but 
onl)5 a temporary one. The "Governor's. Task Foi^ce on Educational Reform" 
had reported the need for a continuing measure of pupil achievement. 
The department staff developed a di'af t bill which provided continuing 
authority for the program. ^Staff were concurrently working on re- 
visions to the state compensatory education^ legislation to include 
eduqational deprivation in the criteria. It was natural that the ' 
draft bill to authorize state assessment as a permanent program would 
tie-in compensatory programs. The draft bill with some minor changes 
was enacted as Act 38 of^ the P. A. of. 197-0, and remains as fhe legal' 
base for tiie program today.. The legislation broadly states: "A state- 
wide program of assessment of educational prpgtess. and remedial 



assistance in che basic skills of scudencs in reading, m^hemacics , 
language arts and/or other genergl subject areas 'is established in 
the department of education. . ^ , 

The provisions then go on to give various elements of the program. 
Included are: "establish achievement goals, provide information use- 
ful for allocation of state funds to equalize education' opportunity , 
provide incentives to introduce programs to improve basic skills or 4 
attainments, and provide the public imformation on the schcol system. 
With this legislation, the state assessment had a definite mandate% 

' ** - * . 

The assessment tests from 1970 vefe revised for use in fhe 1971 
assessment. The tests were lengthened so that each' fourth and seventh 
gi;ade pupil would take a full test battery for reading, mechanics of 
written English 'and mathematics. . The tests would now yield scores 
reliable enough for repor'tin§ individual pupil socres, as well as 
^ aggregate scores at the school building and district levels, ^nd a 
state report. The tests were administered throughout the state in 
January, 1971, along with the st;ill controversial socio-economid status 
and aj:titude scales. ; * ^ ^ 

Before the t^est administration period was over,* a group of local 
superintendents met to review this "new" state program of assessment- 
Tfiese discussions led to acfion by some thirty-height of them. They ^ 
ordered that the test an^^r sheets be held in the district and not 
sent to the scoring service, - The press picked up the story and the 
state assessment became a big story... the program had visibflityl 

After two weeks of unsuccessful discussions where state officials 
tried to convince the superintendents to send in the answer sheets for^ 
^scoring, the State Superintendent and the President of the State Board- ' 
of Education sent a joint letter to local superintendents and boar^ 
presidents, The letter qited "Act 38"Uuthotity for the asse^^i^nts, * 
directed the submission of answer sheets, threatenedf'Qourt action and 
offered to discuss the superintendents* concerns, T^e* superintendents ,^ ^ 
though reluctant to comply, chose not to challenge the state authority 
further. ' * . - - ' - 

In the ensuing discussions local superintendents Vaised several 
issues. The major issue was, of coutse, the intrusion of the state into 
^ local sc'Viool affairs. €ach of the seven or eight meetings between De~ 

partmeht staff And superintendents began with this issue -and required a , 

rejustif icat±an of state assessment and the state authority. Other 
.charges yete iaade, such as, 1) tke tests were invalid and did not 
correspond to the/>lichigah Cijiri^ulum, " much less the "unique" curricula 
.of the many loc^l districts ;^ i) the tests were ill constructed and urvi- 
reliable; 3) the 'test information was no different than that ^already 
known from local testing; 4) local educators had ao opportunity to ji ^ 
-participate in the planning of the state assessment program; and 5) 
reports of r^sdlts would be m^de to the public.^ The criticisms were 
in part acknowledged by the staff and promises were made to be more 
responsive to the involvement and data needs of local educatora. iThe 
technical issues, i*e., reliability and validity of the t^sts; were 
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defended and- final disposition of the charges were left to the publica- 
tion of Dhe technical reports. 

After nearly .seven months of monthly meetings, the superintendents, 
though still ^t satisfies!, decided further discussions were unnecessary. ^ 
They would cooperate' in the future, and the Department would form ^n 
advisory council to' help form the future of state assessment. 

- Public Release -of Results 

The promise had been made by Department officials in the first 
year of the progr^ that the state would not release the "scores"" of 
individual school districts. With this promi^e^, local school officials 
felt secure with the new program and cooperated in the administration 
with a miaimum of' objection. Scores were reported directly to local 
school superintendents and to state officials. 

In the s'econd year, the press, and eventually legislators, made 
inquiry about the "scores" and were told they could not be released. 
This led to a confrontation. An influential legislator threatened to 
introduce legislation which would mandate the release of the state 
assessment data for schools and districts and would provide guide- 
lines for the release. After discussions with the legislator, the 
Deaprtment policy on release of these data vas changed and the basic 
skilly as-sessment data were provided to the press and legislators. 
Even today, ten years later, the promise which couldn't be delivered," 
i.e., no public release of school or district results,. is remembered 
by some superintendents. 

The first release of^ results was made in response to individual 
requests. However, the interest was great and the State Superintendent 
decided to publish. the results for all districts. The first to be pub- 
lished wete the 1971 results. A'compilation of data (assessment test 
results, staffing, financial, dropout, etc.) was made* A book was put 
together and released abc^t a. year after tne test administration; 
ironically the book had a red cover and the press conference for its 
release was on Valentine's Day^, 1972. Superintendents had no love 
of the asseissment, and saw no humor in this^i The 1972 results were 
published in like form. . . the book being browA^ and the release was at 
Thanksgiving, 1972.2^*- The; "thank|" of local school people w*as that 
this was the last book of all district results published by the state. 
The "heat" was too much and the Superintendent decided results from i 
1973 were to be released on 'request, but *nd compilation of all dis- ( 
tricts was released. . *• -J 

After 1973, rank ordering of school districts was not done. The 
tests were 'changed frou) norm referenced, which could be reported succinctly 
in standard^ scores and percentiles, to objective referenced, which, 
were reported in proportion of pupils mastering each objective./ Since^ 
there were over sixty objectives, the publication of even district 

data for all districts was too burdensome. Assessment results , 
e available on request, and often>were listed in the news- 
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The ^(felic release of results was made even more necessary in 
1977 when the Stats'* of Michigan enacted the Freedom of Inforrjation 
Act which required public agencies to make information available upon 
request.^ ' ' ^• 

The policies on release of assessment scores have been influenced 
by educators, legislators, public advocacy groups, and assessment 
technicians. The "pull and tug" to derive a policy involved the desire 
for widespread public disclosure on the one hand^, and the fear of mis- 
interpretation and misuse on the other. School ;feople f^^red invidious 
comparisons of s-chools, and judgements of school effectiveness based 
on a narrow set of lieasures. -Public advocacy groups pressured for 
f;:ll disclose and specialized reporting for subpopulations , e.g., racial- 

. ethnic groups. Assessment technicians counseled caution in generalizing 
from the data and sought ways to provide bej^per interpretive reports. 
Legislators, a.t first, wanted full disclosure, €ut^^nave more recently 
pressed foe Recognition of the ■ limitations on tr^e^data. In fact the 
Legislatijue inserted a statement into the Department budget bill which ' 
prohibits the use of assessment as an evaLuation of schools. The current 
policy of disclosure is to make results for a school or district avail- 

^able upon request, but to provide explanatory and interpretive materials 
along with it. The State Board in 1979 sponsored several worksl^ops 
for local educators and the* press . Tne Boa'^rd's purpose was to assist 
local educators to work with the press to achieve full disclose with 
responsible reporting. Also,* the State Board adopted a policy which 
stated that assessment resultswere not appropriate to use in the eyalua- 
tion of an individual, teacher . ^ . " . 

Evaluation of 'Schools ' . ' 



Tied closely as an- issue to the public ireporting issue are the 
issues of use of assessment results in making decisions. How good is 
my school or district has always been thf pfime question of interest 
to citizens. Before state assessment the judgments undoubtedly were 
made on criteria ranging from hearsay, to athletic teams, to -tfiej number 
of graduates , getting scholarships, to any of numerous other^lactors . 
State assessment was of interest ^as a cijiterion for judgi^rg^he worth 
of a school or district because it was reporting on how well pupils 
were ^learning. After all, schools existed to teach the basic skills 
j^ftd should be rated on how well this was accomplished. Mewspapers 
Reported scores and pointed to high scoring districts (mean of pupil 
sc^es in the district was used, and \ater tjie proportion of pupils* 
mastered more than 7?% of. the objectives r^ested) as "good," and 
low scoring as "poor." Real estate agents, too, tried to u^e the 
scores, if it suited their purpose, to feteer customers to buy ,in "good" 
istricts. 

The- comparison of school and districts on ass^essment scores alone 
v?as a concern for school administrators. They carried their dissatisfaction 
o some key legislators as well as State Board members. Under pressure 
from the legislators, the Department initiated a large campaign to assist 
local educators in the proper and full reporting of results. A<ivQcated 
were early reporting, and reporting in the context of other information 

ft 
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about education, i.e., the financial, staffing and otber conditions 
of education. The idea was to put the assessment scores in a larger 
context to provide for a fuller understanding and a better "evaluation" 
of the schools than a simple judgment made on one set of test scores. 

Maximize Policy Use or Instructional Use 

Initially (i.e., 1970-73) the state assessment program used norm 
referenced tests developed^ by a testing company from existing items to . 
the Department specifications. The primary purpose of the program was 
to measure the status and progress of basic skills achievement in the 
state and its districts. These tests ia reading, m^ithematics and 
mechanics of written English provided d^fe for these purposes with a 
minimum of expanse and testing time. An' aggregate achievement "score" 
for the state in each area was computed, £s was a "score" for each 
district in the state. Districts were easily ranjced by percentiles, 
and districts in need of assistance were ^sily identified-. State 
■^Dolicy purT^oses were well served by the use of norm referenced tests. 

t 

Politically .though there was discontent with, the program. The 
discontent involved: 1) the use of the results to compare school 
districts, 2) the tests did not provide information useful to schools 
in instruction, and 3) the tests were not "Michigan tests" and Michigan 
educators h^d not been involved in creating zhem. 

After the furor created by the superintendents in 1971, the State 
Supetintendertt decided to both be responsive to the issues raised abput 

norm tests, and to exercise state leadership in basic skills curricula 
for the state. It was decided to change from norm referenced to objec- 
tive referenced tests for th^ state assessments/ The decision would 
switch the emphasis to maxim!ft"H^he instructional and curricuiiim uses 
of the results, at the local level, rather, than the policy use at the 
^state level. ^ 

The State Superintendent met with each of -the statewide curricu- 
lum organizations (i.e., mathematics, reading, science, social studies, 

'health education, physical edi^cation, art, music^ and challenged them 
fo specify the basic expectations for their area. The basic expecta-- 
tions were, in general, defined as what every pupil should be able to 
do and shoulji know at the end of gr^e 3,. grade 6 and grade 9. These 
were to be "minimal expectations" for all pupils in Michigan schools 

^ and would be s-trongly advocated as the minimiim curriaula for all 
schools in the State. The curriculum organizations, after much dis- 
.^us'sion, all chose to respond and work with Dep^artment curriculum 
specialists to s^cify the "minimaL expectatiotls . " 

During late 1971 and early 1972 the curriculum specialists drafted' 
the expectations. These were reviewed and, in some caees, revised by 
committees of generalists (i.e., teachers, principals, school board 
members* school administrators and par^ts). Finally, in 1972 the 
State Board adopted the filrst two sets* of expectations, or objectives. 
These were the reading and mathematics objectives which were to be 
used In the new* state assessments. 
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The tests were constructed from the objectives. The Department 
engaged some local school districts to provide: 1) teachers to vrite 
test items, and 2) classrooms for test tr^uts. The tes^:s were to be 
written primarily by Michigan teachers based on Michigan produced 
objectives ... these were to be '^Michigan tests.'* The Department also 
contracted with a testing company for support services to insure that 
the new tests would meet high technical standards. 

The tests were completed and^ready for use in the fall, 1973^;,.x-^ 
The test administration) time was" chafed from January to September- 
October with the initiation of the objective referenced program. This 
was done because of the emphasis on rnstructional uses, Thfe early 
administration , allowed the return of results early so that individual 
pupil needs could be identified and t^eachers would have time during ' 
the school year to p^rovide remediation, if needed. The reports wou] 
identify the objectives mastered, and those not mastered. Master^/ 
defined is answering correctly at least four of the five test items 
^^or each of the forty, mathematics and twenty-tjjree of the reading 
objectives. The reports contained detailed information compared to 
the general information contained in the norm test reports. The detail 
of man^ scores made it more difficult to compare schools ^ad districts 
on the basis of state . assessment , but made the inf onnation#fore valuable 
to principals and teachers. 

The State- Board had* used the state assessment program to exercise 
Leadership in Michigan education. For the first time a common curriculum 
hacMfee^ specified, albeit it was only a minimal level and was suggested 
tather thaft mandated. The minimum expectations though were to be^come 
useful in promoting equal educational opportunity initiatives in schools 
of the state. 

State Level yses 

^ Tne change to objective referenced tests and the more detailed 
reports was responsive to local education criticisms. Hdwever, when 
the fiirst reports were released, the press, atvd state officials were 
confused. by the many figures. ^ Tney wanted^ to *be able to tell whether 
or not schools were, doing better than, last year,, and.^hich were "good*' 
and which were "poor'* achieving schools. There was a demand for. a 
simple summary, type' report. The State Superintendent asked foj a 
single score, . , ' 

the political 'pressure from State officials led to the development 
of a summary type report. The report was added in 1974 and was called 
the "proportions report." The report gave the percent of gupils 
mastering objectives in each of four categories (i.e., 0-24, 2d-49, 
50-74, 75-^100 percent). The reports were in reading and mathematics 
and were produced for schools, districts and the state. The fewer 
figures were more understandable and useful to l^pet;sons and for 
state purposes. - . , i 

The proportions reports were used to set criteria for identifying 
* levels of needs in Michigan schools (e.g., schools with fewer than 



50 percenc of che pupils mascering 75 percenc or more of the phjec- 
cives were defined as high needs schools). The Scace Superincendenc 
and staff directed special assistance to these schools in an effort 
to assist them to improve. 

The assessment program reflected an action policy of the Depart- 
ment of Education to seek and use information to develop a concerted 
program for educational improvement. However, it was too general to 
provide information to assist in determining the success or effective- 
ness of specific program efforts. Thus, concurrent with the develop- 
ment of the assessment program, the evaluation program was also being 
developed, ^ * ' ^ , . 

Initiating Program Evaluation 

As noted in an earlier section, the advent of e:ictensive federal 
involvement m education, in particular, the Elementary and Secondary 
Education Act of 1965 (ESEA) , provided d new impetus for State Educa- 
tion Agencies across the nation. In. addition to creating an expanded 
role for State Education Agen^es in educational program development 
and adcjinistration, ESEA demanded a more active role for State Educa- 
tion Agencies in evaluating those programs. 

ESEA caught most state and local education agencies unprepared 
to undertake sophisticated and technically sound program evaluation. 
Federal officials were vague in providing direction and frequently 
suggested summative questions which needed response. These questions 
were usually descriptive as well as sunmUtive in nature. Beyond 
these basic descriptive questions. State Education Agencies were 
encouraged to develop evaluation capabilities and design %valuations, 
to best %Qeet the needs of state and local constituents. 

In Michigan, as in most states, early evalupfion efforts were 
aimed at meeting the summative evaluation requirements . The prevailing 
philosophy wasi that evaluation was a federal reporting requirement 
which had to be done in order to maintain eligibility for funds. These 
"reportijig*' activities were decentralized in the ^tate agency a^part 
of the 5^/erall responsibility of, the persons who administered the 
programs. The evaluation results were seldoi? used (nor thought to he ^ 
useful) in program administration 'or policy development. 

The decentralized approach and the "required reporting" philosophy 
toward evaluation began to change in ^^ichigan in 1969 with the creation 
^ of a 3ureau of Research, With the establishment of a new Bureau, came 
the direction for che new ^taff to begin conducting evaluation of the 
new federal programs, and to use some of the federal money to support 
these evaluations.* This new commitment was further strengthened by^ 
the appointment of a new Superintendent who, as noted earlier, believed 
that information provided by technically sound evaluations would lead 
to improved decisions regarding^ educational program^. 
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The early and active support or the State Superintendent resulted 
in a decision to begin recruitment and employment of a small number of 
specialized staff and to begin centralizing the function of evaluation. 
Evaluation staff were to ber administratively independent of the person- 
nel who were, responsible for management of the programs to be evaluated. 
The new staff were asked to develop and implement a systematic approach 
to program evaluation. 

• ' * A 

In 1974, the" State Superintendent emphasized his support of 
evaluatxpn and put the full weight of his office behind the centrali- 
zation of the evaluation functions in the ,evaluatidn program, was issued 
Portions of the policy statement follow. > 

It is my intent that all evaluation activities^ 
sponsored by the Department of Educatioh be co- 
ordinated by staff with expertise m evaluation 
so as to maintain con^istencv in the evaluation 
errorts.-" 

« j ' . , 

' After citing several negative aspects associated witn decentralized 
evaluation efforts, the Superintendent's statement continued: 

If evaluation is wo^th doing, it is worth doing 
^well. Furthermore, program administrators srioul^ 
never evaluate their own programs. Therefore, 
effective immediately, I am" asking each of you to 
ensure that the evaluations of^your programs are 
coordinated through (the 5vaiuat:^gn Program) which. *. 
is responsible for evaluation." 

J ■ - . v. 

Th« statement coaeluded by indicating actions wnicn snouxa De 
taken to receive 'ap^rfoval for evaluation activities. 

In 1977, the State Superintendent repeated h::^3 statement verbatim 
and added that he "expected any items which included plans for evaluation 
such as programmatic state *^ans, to include a statement of support 
from the evaluation 5Xaff before being suuMm^ to the State Board of 



ilducation for approval .^^^i^ 
The Evaluation Process 

The goal of evaluation was to provide inforjnation to educational 
decision-makers so program impttrovements could be made. Staff were 
committed to the task of demonstrating that well-designed, carefully 
implemented and properly supported evaluation "prqvides objective 
information for planniitg, administering and improving educational 
services at all levels of educational governance, from federal and 25 
state to school district, to 3chool building and to classroom levels." 

In an enterprise so large and encompassing as education with so 
many factors beyond the control of* the evaluation specialist, it is 
'impossible to employ the same experimental rigor which aright be found 
in ^scientific laboratory, In the social science? and education it 



is often impossible to control conditions and set up experimental 
designs as in the natural sciences. Therefore, the evaluation model 
employed by the Department had three stages: (1) descriptive evalua- 
tion, (2) evaluation to determine success, and (3) evaluation to 
determine effectiveness. Descriptive evaluation refers principally to 
the quantitative description of resources (human, financial and material) 
and purposes associated with educational services. Evaluation of 
success refers to quantitative and qualitative judgments regarding 
whether or not objectives of an educational delivery system have been 
met. Evaluation of effectiveness ^refers primarily to identification of 
factors associated with success and the relative costs of assuring that 
those factors exist. \ 

While these stages are sequential in nature, they are also fluid 
and overlapping. For example, it will take some evaluations a 
mdre to pass through the descriptive evaluation stage while others will 
pass through this stage much more quickly. Also, work may be "occurring 
in more than one stage simultaneously, for example, evaluation of success 
may begin before the descriptive evaluation is complete. Furthermore, 
the implementation of each successive stage does not mean th^t the prior 
stages are terminated. Rather, each successive stage builds upon the 
information provided by the preceding stages. 

In the^last half of the 1970s several evaluations were able to 
identify factors which are related to success. Based, in large part,. 
»on these evaluations the Department is now exploring means by which, 
through a state-local partnership, strategies can be developed which 
will lead to more predictable program improvement. , 

y fi 9 7 

As part of the State Superintendent's policy stat^eiaent, ' " he 
asked program administrators to enter into ^'Agreements for Services" 
with the evaluation staff for the conduct of evaluations. The "Agree- 
ments, for Services" reached between program administration staff and 
evaluation staff specify ser«/ices and responsibilities of bot*h staff 
and formalize expectations of both parties. The agreement commits the 
a/dministrative unit to provide a mutually agreed amount of f'onds to, 
conduct the evaluation. The agreement commits the evaluation unit 
to provide information to answer specified program and policy questions. 
Staff employed ^or the evaluation are administratively and progranznati- 
cally .independent of the program administration staff. 

While administrative independence is desirable, daily substantive 
interaction among the evaluation and program administration staff is 
essential. Evaluators must be aware of, and sensitive to, the subtleties 
of the program they are evaluating. Alsb, informal substantive contact 
decreases the threat oftren associated with evaluation. 

Much emphasis is placed on cotmnunication among staff and the 
appropi*iate use of evaluation results. In addition to the day to day 
contact among staff, more formal mechanisms are used for presenting 
findings and r^ommendations . Formal "exit conferences" are attended 
by evaluation staff, program administration staff, and, frequently, 
one or more high level officials of Uis^epartment . At these "exit 
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conferences," findings of th'e ^ evaluaxion and accion-orienced recom- 
mendations are forinally pres;ented by evaluation staff. Program admini- 
stration staff respond, either at the "exit conference," or soon after, 
regarding actions" they plan to take on each recommendation. The "exit 
conference" reduces the likelihood that evaluation finding's and recom- 
mendations will ignored, the administrator in charge of both the 
program and evalu^tion^ responsibilities is present, supportive and can 
direct actions and polity responsive to the evaluation findings. 

The State 3oard of Education has, historically , "been very interested 
in the work of the evaluation staff. Great care Is taken to prepare and 
presetit evaluation reports to the State Board which will be meaningful 
as a Cool ro guide in establishing policv. Frequently, major segments 
of time are get a,side by the State Board of Education 'at Committee of 
the Whole meetings to discuss evaluation reports, recommendations and 
implications for adminis tractive and State Board action. 

. ' , 

In addition to fonoal and informal 'communication efforts with 
Department stafs^gd the State Board of. Education, evaluation staff 
are actively ii^^Bed in a program of tethnical assistance and dissemi- 
nation to ioca^^raucation agencies. These activities cross a broad 
range rrom disWibution of executive summaries of evaluation report p 
to formal inservige or technical training sessions. 

The Conflict Between Expectrations and Methods " ^ 

There has been a strong policy atid programmatic commitment to 
evaluation in the' Michigan Department. An environment has been 
created to promote the development of a strong organization for con- 
ductings evaluation. Tne .contmitment to use evaluation findings and act 
on evaluation recommendations has been high on the part of tSie staff 
of the Department and the State Beard of Education. But even in an 
organization with this high l^vel of commitment conflicts between 
expectations and methods can and do occur. 

I 

One of the most common areas of conflict is a conflict among 
program priorities. These conflicts are primarily of two types. One 
type of conflict is a result of a lack of clear enunitation of the 
purposesjand priorities of a gi^^en program* Evaluation staff, during 
the "desTriptive stage" of evaluation^ work with program administrative 
staff to clarify the purposes and objectives of the program to be evalu- 
ated. This process can be a particularly frustrating one, if there 
are external pressures to provide information quickly. The lack of 
^lear enunciation of purposes and priorities becomes acute when the 
evaluation effort begins a lengthy period of time after the program 
begins. 

k 

The Examination of program purposes and priorities frequently 
leads to a second type of conflict among program priorities. This 
occurs fwhen the programs have a mixture of social action and educa- 
tion priorities. For many reasons, categorically funded programs 
often have a multiplicity of apparent purposes; some establish 
primarily education priorities while others establish primarily action 
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priorities. For example, legislation may contain language' which seems 
to equate civil rights and basi^e~-skills education. 

It is not uncommon for these social action and eatication priorities 
to be so closely intertwined that it becomes virtually impossible Co 
distinguish among them. The "descriptive" stage of evaluation is used, 
to deal with this problem (as a part of enunciation of purposes and 
priorities). However, even if the social action and education priori- 
ties can be identified and separated, these programs are especially 
difficult to evaluate. In some cases, program administrative staff ^ 
have preconceived expectatLions regarding the outcomes of an evaluation. 
Additionally, they often do not fully understand that social action \ 
objectives cannot be measured by educational performance measures. 
This combi^ftion of ^econceptions and misunderstandings can lead to 
great disappointment upon the completion of the evaluation. 

Evaluation must be especially careful to develop mutual under- 
standing about purposes, priorities and expectations of the program 
to be evaluated. Of equal importance, is the development of mutual 
understanding regarding expectations for the evaluation effort. Eval- 
uation staff of the Michigan Department of Education use the "descrip- 
tive" stage of evaluation to develop these understandings. However, * 
the affirmation of these understandings must be continuous. 

A second area of common conflict betr^een expectations and methods 
is ^he. conflict in requirements. This type of conflict is the conflict 
between program funding mechanisms and expected results, and most 
often occurs when .progr^s are funded on one ^et of criteria and the ^ 
program success is judged on a different set of criteria. For example, 
the funds are provided for reimbursement of program staff salaries, 
but tne ' evaluation ^ocus is on now much the participants achieve. 
This particular Cype of conflict may create hoa^fcility among local educa- 
tion agency st.af f who feel that it is* unfair to conduct a state-level 
evaluation of those parts of the program funded locally. 

This type of conflict often establishes a negative political environ- 
ment within which it is very difficult to conduct an evaluation. Evalu- 
ations fraught with this type of conflict usually will not advance pa,st 
the "evaluation of success stage." The evaluators can use the earlier 
stages of the evaluati^on to establish reasonable measurement criteria * 
and data collection procedures.. However, there are likely to^be so 
many negative factors beyond the control of the evaluators, that some 
such evaluations will never leave the "descriptive evaluation" stage. 

Another area of conflict occurs between federal constraints and 
state-local policy and program needs. Historically, the requirements 
of federal programs have focused on sunmiary reporting and are of little 
ose in state or local program or instructional decision making. This 
is not a' problem so long ^s state and local education agepcies are a^e i 
to exceed these requirements. In fact, if tRe burden of federal jre-^ 
porting is minimal and federal funds can legitimately be used to expand 
^the evaluation of' that program to yield results meaningful to state 
and local Educators, a positive state of affiars exists. The conflict 
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occurs when federal requiremencs, even chough sunnaacive in nature, 
are so burdensome chac all of che resources available muse be used 
^in meeting, Che fede^ral requiremencs. A review of che hiscory of evalua- 
cion of ESEA Tide I in Michigan snggescs chac chis paccem has occurred 
SSEA Tide I will be discussed in more decail in a lacer seccion of 
chis paper. 

Recencly, a second type of conflicc becween federal program re- 
quiremencs and scace-l^cal policy and program needs for evaluacion 
has become common. Federal programs are becoming more and more pre- 
scripcive vich x^ard Co mandacing specific informacion which muse be 
gachered and the specific evaluacion procedures which muse be used by 
scace and local evaluation staffs. These procedures are frequently 
insensitive to state a«^ local policy and program needs. Further, the 
constraints are such that states can do little to design evaluations 
to meet both federal requirements and state and local needs. An 
example of this "federal prescription service" are the rules and regu- ' 
lations dealing with data collection and evalij^^tion of programs funded 
under ?L 9^-432 (Vocational Education Amendments of 1976). 

?^ 94-482 and its associated 'rules and regulations requxre both 
evaluation and reporting of management data (the Vocational Education 
Data System — VEDS) on every vocational program.' Tne evaluation require- 
ments, by themselves, are manageable and considered by many to be useful 
However, the reporting requirements of VEDS are so prescriptive and 
burdensome chat the Council of Chief State School Officers (CCSSO) has 
officially opposed them and threatened^ to refuse *to comply. This 
enormous data collection burden imposed by the federal government has 
made it very difficult for evaluators to collect and analyze new data 
needed for more meaningful evaluations. 

A third general area of conflict is a conflict in commitment. 
Frequently, top level policy makers do not provide adequate support 
for evaluation activT.ties because they have an incorrect impression 
of what evaluators do. This is especially true if -t^e only visible 
product of the evaluation effort is an annual summary report which has 
little perceived Osefylness. Evaluators need to do a much better job 
of helping policy makers understand what evaluators do. Formal and 
informal communications must not stop with program administrators.^ 

Another problem is that commitment from top level policy makers, 
as evidenced by resource availability, is inconsistent. Ironically, 
in periods of economic difficulty, resources-^or evaluation may 
.actually increase as decision makers seek data to help with the manager 
ment of decline. In^ economically good times, evaluation may not seem 
as necessary and resources for evaluation may become less plentiful. 
This inconsistency, even in a state with a generally high level of ^ 
commitment, makes ^ong range planning for evaluation somewhat more 
difficult tihan desirable. 



A Case or the Influence of Evaluation on Policy 

Michigan has had a state compensatory education program since 1967. 
The first years of the program funds were distributed to school dis- 
tricts as formula grants. The formula used economic, cultural and 
social factors for defining educational need. However, beginning in 
1970 Michigan began to define educational need in terms of pupil 
achievement, i.e., a direct measiire of educational need rather than a 
proxy measure. The state compensatory education program makesyuse of 
the state educational assessment results for this purpose. The' pro- 
cedure for determining school district eligibility, beginning in 1971, 
for the compensatory funds was: 

1. Pupils scoring below the 15th percentile on the 4th 
and 7th grade state assessment test were defined as 
pupils to be counted as ^'eligible." 

2. The proportion of all 4th grade pupils deemed ''eligible'' 
was computed; likewise the proportion of the 7th grade 
pupils. 

3. Applying the proportion of eligibles in grades 4 to grades, 

an estimate of "eligibles'' in those grades was com- ^ 
puted; likewise the grade 7 proportion was used to estimate 
grade 5-6 "eligibles." ' 

4. The school districts were ranked (high first) according 
to the proportion of "eligibles" in the district. 

5. The district allocation was computed by multiplying 
the number pf "eligibles" in grades K-6 (the State 
program was limited to these grades) times $200 (the 
funding level). Districts were funded in rank order 
until the total State appropriation was used. 

It was determined that the 5tate could .afford $22.3 million for 
the program before the formula for the ^program was written into legis- 
lation. Legislators used several computer simulations, each with 
different eligibility and/or fund level criteria, in the p'rocess of 
setting criteria. Basically the data were used in making political- 
policy decisions. Legislators wanted to know which districts would 
be funded, or not funded, and at what level, before agreeing on the 
formula. The final formula was a compromise made by members of the 
appropriations and education committees of the legislature. 

- The use of an achievement test indicator for determining the level 
of educational need was But one of several different features of the 
Michigan program. Others were.: 

Assurance of three years of funding once a district 
was deemed eligible . 

2. Provision for funding adjustments- based on program success. 
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3. Provision for annual evaluation of each pupil *s. progress 
to determine level of Attainment. 

4. Provision of considerable local discretion vested in 
local districts in the use of funds. . ♦ 

Department staff worked together vifh local educatorsV^o design 
the program. ^The local educators were interested in three things in 
the new compensatory education program: i) more money, 2) more dis- 
cretion in the use of the money, and 3) greater assurance that the money 
would be available fot nore than one year. Each of these was attained 
in the new legislation. 

• 

The State in this program was interested in two important proposi- 
tions: 1) could schools be -held .accountable for educating the lowest 
achieving pupils in the schools, and 2) cov^d additional money for 
^ basic skills instruction result in higher pupil achievement? 

The program has changed over the years since 1971. When the State 
assessment changed to objective referenced tests in 1973, the criteMon 
ror eligibility was changed from students below the i5th percentile to. 
students achieving fewer than 40% of the objectives C^st^d. Again the' 
legilsature used^ several siinulations of data to set the formula so tha't 
the funded districts and the funding^ level remained comparable to the ^ 
previous years. 

^ The three years funding feature was later changed to an anjiuAl 
redetermination of district eligibility and -funding levdl. Non-funded 
districts lobbied for the change so they would have a chance for funds 
before the three year cycle was completed. ^ ' ^ 

A philosophical belief was tharall pupils regardless of race/geo- 
graphical location, economic status, etc. could attain' basic reading 
and mathematics skills. Thus, a feature to reduce funding if pupils ^ 
didn't achieve was included. The adjustments were to be made annually 
on the basis of pupil achievement. Pupils* who achieved at least 7.5 
months gain, as measured in grade equivalent units oh a standardized 
test,^ received a full $200 allocation for tte next y«ar. However, if 
achievement was lower 'than 7.5 months, the cistrict received a lesser 
amount (the proportion faein^ the gain in mor|ths to >. 5 times >^200). 
Local districts accepted this feature initially to get the money. ^ 
After the first year they lobbied to retain the money which was going 
CO be ''lost" because '*the kids still need the help." The money was 
reallocated to the distrfct provided -they f iled^ a plan^to meet the 
needs of the students who_were still low achievers. After two years, 
and threats of losing more mon.ey, the districts succeeded in getting 
the legislature to delete this '"accountability" feature from the program. 

The evaluation of tM compensatory education pi^ogram was linked 
very closely with the "accountability" feature. Since the funding was 
determined on a per pupil basis and that was tied to level of attain- 
ment, it wa^ necessary to evaluate on an individual pupil basis. State 
guidelines called for*a pre and post test (either spring to spri-ng, or 



fall to apring adminis.tration) \using approyecf s.t^ifdardized c^acs. 
Scores for each pupil were subm^.cted co**che Sc'ate and were used both . 
for program evaluation and for the determination of funding^ The veri- 
fication and processing of over 112 ,000* pupil records was 4uite a ^ 
challenge for State evaluators. ^ ' ' 

The evaluations of the program showed*. the progi:^m to be a success. 
The districts committed themselves to developing quality basic skills ^ 
programs based on specific perfoimai>ce objectives. Strategies were 
^devel^ped to provide services to low achieving ch'ildren regardless of 
s the school attended, ti^us moving away from the "target" school concept. 
Most important .of all, the program resulted in improved achievement for ^ 
DUDils in the program. 

The State evaluators not only analyzed data from the scrfool dis- 
tricts to determine ptpgram success, made recommendations f or'*'progi>am 
improvement and provided funding allocations, they also used the pro- 
gram to improve evaluation techniques ae^ros^ the 'State. The evaluation 
of individual achievement , as**Vell 'as program evaluation, presented 
many local sc^%>l educators a challenge beyond theip^nowledge and 
skill level. Jtate staff were able to seize this Opportunity to provide 
^.nservice trainiiv^ to improve evalUatiop pethodology and , data use in ^ 
m^ny school districts. Particular emphasis was placed- on working with 
local district staffs to' develop objective-referenced tests for evalu- 
ation purposes. * 

Unfortunately, the elimination of th^ funding adjustments based 
on the success feature made some people believe there was little need 
to continue prpgram evaluation* Thus, funding to maintain ^the State, 
- evaluation staff was deleted from the Departaeiit budget, even* though 
for another tharee years^the mandate to provide the legislature with an / 
evaluation report remained Cn the act. Local schools, in most cases, 
continued tli€"^f ogram "^Valuat rofr"afld~ u5"€d "Th e resTrits^^irxraiir^,^ -irawever ,- 
State activities stoprpeti with "the withdrawal of funding. 

w At .the time evaluation, funds were deleted, it was suggested that 
^^--^tat6 assessment results be used to^ evaluate the program. The belief 
\ was, since State 'assessment was used to determine eligibility for com- 
. perisatory education <unds , the same test, over the same objectives, 
•should be used for evaluation. Very simply, it was thought the fourth 
grade results Vould show success through the first four years, and 
seventh the last three years of the program. There was "a certain logic ^ 
to the proposal, however, there were many falacies: 1) State assess- 
ment results were reported for the total pupil population, not com- 
pensatory pupils as a subpopulation; 2L pupils moved in and out of 
compensatory prpgrams and treatments varied; 3) one measure was not 
sdfficienC to evaluate program su(?ce'ss and would tell nothing about 
• why some programs succeeded more than others. 

q^e proposal vas hotly debated. i^ 'qheyn)e^partment, and an attempt 
was made td implement th^ '^'evaluation" "By identifying individual pupil 
assessment data with the compensatory services* the pupil received. 
' lK)cal school administrators *and evaluators s|:rongly opposed such 'fevalua- 
; .^on." . ' ^ ' 
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^ - . 

Th^ local staffs, refused tp cooperate in the coding and the ^ 
attempt failed. After negotiations, it w^s deci(led to u$.e state assess- 
ment as a vehicle for coliedting some data about the compensatory pro--' 
gram.- Local evalua^tors agreed -to code pupils on the fall assessment as 
enrolled in the various compensatory programs, i.e., ^ESEA Title I/Stat^, 
'Bilingual. ^This allowed the State'^co address some questions with^policy 
.implications: 1) were the lowest achievers in compensatory programs, 
and 2) were pupils enrolled in more than one compensatory p^^gram. 
In- case the lowest achievers were not in the program, more sfringent 
guidelines for pupil selectioi^ could 'be imposed. ' Data for the second 
question would be used to address whether or not the greater benefit 
was to' continue multiple funding or spread nioney to new pupils. 
.ct)ding project is now ia its first |rear 'and data are being analyzed. 
As for the more in depth questions ^f evaluation, local districts and 
the Department are cooperating in special case- studi-es . These studies 
will .address the reasons some programs are successful and. others are not. 

me State Compensatory Program is a good example of using data to 
carry outja philosophical belkef, i.e., low achievers should be benefited 

^^S^^ ' ^ ^ a good example of evaluation being used in the 

mana^ment of the program and its improvement. Unfortunately, the ^ 
ev;2Quation efforts were not appreciated and resources were withdrawn . 
p^lpr to the potential benefits being attained.* The program continues, 
bJc there is no way of systematically judging its . effectiveness 
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A Case of Categorical Constraints 



The preceoThg discussion of Ai^ticle 3 evaluation has presented a 
case stQdy of evaluation's impact on policy. The evaluation of ESEA 
Title I is a' case study of the categorical constraints on the usefulness 
.of ' evaluatix>n in policy development. 



, Tide I evaluation development paralleled, in many respects, the 
^valuation of Article 3. In fact, this federal program, more /than any 
other single program, provided impetus for evaluation in ^tne Michigan 
Department of Education. 

r 

> - • ' / * ^ 

In the late 1960's and early 19'70*s, the federal reporting re- 
q'uirements for ESEA Title I were quite minimal. Summative informatioi\ 
Qf a descriptive nature was expected and achievement data. were desired 
• by-^ederai officials. However, it was recognized that many state and* 
local* education agencies did not h-arJ^e the capability to conduct more 
sophisticated evaluations^.% Federal officials encouraged 'State and 
local officials to develop capability co do evaluatio^is^ that exceeded 
the minimal expectations. 



Thus, duAng, the latk 1960*s and early 1970*s*the evaluations of 
^Title I conducted by the Michigan Department of Education were almost 
entirely descriptive, consisting initially 'of baseline infonnation) such 
'as'Tiumber of, students, number of teachers, and amount of money spent. 
Beginning in 1972, the evaluation started to yield accurate and useful 
infonnation regarding succ^s (in terms of achievement of students) of 
the program across the State. This information was based on district 
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level Inf onaaxion. Through the 197jA-75 achool year the design remained 
.relc^tlvely constant 3.0 as, to verify findings of . success. 

-State and federal officials were able to say with* considerable cop- ^ 
fidence that Title I in Micliigan was successful. However, the "success 
evaluation'* did not provide sufficient information to enable state and 
local officials to identify or select specific strategies associated 
.with- success , These. might be used for improving local projects which 
were not successful. Conseqtiently , since 1975-76, the evaluation of 
ESEA Title I in Michigan has focused ory the building as.,<^e level for 
data collection and analysis. ' . . '\ 

\ 

In the evaluation of Title I, the Department of Education evalua- 
tion staff, have been successf-ul in identifying a'^number of variables 
related to ^success. Further, these variables have been verified, by 
other Studies. Thus, evaluation 0/ ESEA Title I in Michigan has 
advancedinto the "evaluation of effectiveness'* stage discussed earlier 
in this^aper*. 

♦ * 
In addition to the progress in increasing the sophistication ox the 
^itate level evaluation, evaluatibn' staff have worked with local educa- 
tion agencies to make local evaluation efforts more useful. For many 
of the same reasons the assessments progr^^ switched to objectiva re- 
ferenced tests, local districts i^re enaouraged to develop objectives 
and -ob^-ee&i^^-^referenced tests for evaluation purposes. 'Together, 
State and local officials worked to make objective-referenced tests 
u*SBful both for instractiqn and evaluation. ' Jn order to assure high 
standards of quality for locally developed objective-referenced tests. 
Department staff produced a quality-control system for objective-re- 
ferenced test;s which was generally followed by local districts. ' 
Locally developed tests had to meet the standards of this quality 
control system before they were approved by the Departments for use in 

"evriwn:oir"6T"TSE?r; -^"TTt "^^7" ' ^ \ ^ ^'^ \^^^^^ 

The envelopment * and planned use o5 00 jective-^ref erenced tests ror 
Titlp I evaluation peake(;^in 1974-75 and 1975-76 4t ab^out the s^m^ time 
that the state. level evaluation was reaching the "evaluation of effective- 
ness" stage. Thus, the Michigan Department of Education Was able to 
take advantage of the impetus provided by Title I to develop sound 
evaluation procedures which yielded 'meaningful results for policy de- 
velopment and program improvement at both the state and local levels. 
Additionally, the data provided to the U.S. Office of Ediication about 
Title I In Michigan was. of high quality.' However, not all states had 
developed a high degree of sophisticartion*^ an^-^^hose which had done so 
had used .different methods and procedures. In sfrort, the federal 
policy of encouraging development of evaluations which were useful 
at State and' focal levels had resulted in data at the national level 
which were not comparable and of varyjn^ degrees o^cyjality. 

In testimony during the debate Reading to reauthorization of ESEA^ 
Title I in 1974, Congress expressed cpina^iderable dissatisfaction with 
the lack of comparable data to guidfe its d^iiberatioas. This dissatis- • 
faction wals specifically exhibited in the Education Amendments 5f 1974^ 
and the Education Amendments of 1978. * ' ' 
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The legis.Iaci^n required the* L\S. Office of Education co deyelop 
and provide Co scace education agenciesi "njodel^ for evaluation qf aj.1 . 
prograps""^^ funded' under Title I "...to be utilized by local e'duca^ 
tioaal agencies, well as by the state age>icy in the evaluation of 
such programs. '^-^^ The law further stipulated that the models shouM^-'' 
yield data Vnidh are coraDa*cable on a state and national basis ^ 

At the time that *U.S. Office of Education began initial develop- 
ment af these . evalua tion models, Michigan Department of Education 
staff were struggling to develop procedures for ag^gregatiti^ data frpn^ 
locally developed objective-referenced tests^ It was hoped that the 
new models would recognize the .value of objective-referenced tests and 
that a sound *model for their use would be developed. It was soon 
learned, however, that sj^jch was not to be the case. 

It became obvious that the models being developed would be of 
limited usefulness to the Michigan Department of Education. Ccy\se- 
quently, while supporting the need for nationally comparable data, the 
evaluation staff of the Department actively advocated the development 
of more flexible models aimed at identifying variables associated 
with*. achievement and greater utility at the itate an^local levels.-^'* 

In October, 1979, the fin^l rales and, regulations"^ were passed/ 
m^^dating- the use of • three evaluation models. The models are puch / 
more restrictive- and less useful at the state and local levels, than ' ' 
hoped for by evaluatioa staff .of * the Michigan Department of Education. 

ihe immediate efrec't has been a retrenchment. Resources are not 
being spent to develop procedures to comp^^y with th$ mandat.ed federal 
reporting requirements. ^'Evaluation of effectiveness" has been side^ 
"tracked and extensive new development of objective-referenced tests 
has -ccme to a' virtual^ halt , in the e,val\fa tion of Tltl^e I .irb^Mi.CXli^ail,^_-. 
It remains to be seen*whether new state and "local uses of the ^nandate^ 
evaluation models can be developed thereby reducing the conflict 
befJeen federal 'cons traints and state-local policy and , program needs. 
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• The role of the State. Board of Educadlon in Michigan changed 
dramatically in tjie last ten ye'brs. The transition was from a rather 
passive presence in Mich^.gan education tol an assertive leadership role. 
The leverage for- the change was hx large fcart because bette.r data 
at/out ^the educational systen ^ecitije available to them. 

< ' ^ ^ \ / " . ' • * 

The stacevfke educational assessment program was initiated in 1970. 
•Th'fe data from^he ^ssessients were used is indicators of the weaknesses 
and ^trength3 in basic skills education J and to monitor progress of the 
, schools and districts of the stace. Thei^tat-e Board based policy i?iit£a- 
.tives ih^compensatory education, a^&buntkbilifcy , eq^jal education oppor- 
tunities, and Q^partment 5er»/ices to districts on in^ oration produced 
by the stite assessments. The program 'decame the. "center piece'* of 
el^ementary and secQndary education in tHe state. 

; 

: Th-e assessment .data were good indi|atot3 of needs, but w^ere of 
Very limited use in providing direccionlln dealing witri the needs. 
;A aore indepth evaluation of programsi was needed to identify "what worKs*' 
/to produce, a betper educational system fand rHLgher achievement for 
children, youth , and adults. "The State [Superintendent , in recognition 
of this, centralized the evaluation fuiiccion in the Depart:ment of pduca--. 
ci'on, and over the years was most supportive of their work. The evalua- ^ 
'tions went through three -phises; descrtutive, success and effectiveness, f 
Especially in, dc^^ensatory Education, /tne evaluation data were important ; 
ih is^t^i^^n:^. df| 'ressoyr.ee a'^^cation, nro^ram management, and policy 
dev^lorartent .', -^--^ — ^-^-^.i^l.^i ^ / ^ _ j^^;..^ ^ . , __ 

'problemfej the 

problems . | ' I \ \l 

^ Jll^QiS5P-^f-^Ji??t?-5P j bme to ip pj^ ^<^^^^ f ' the powljg of dat a I n ' 



WheteaSp the 'Assessment /data drc^ided an indicator of 
evalu^^W daVi providep the dktk for addressing the 



Uecision makihg. TheJjState foard State Sujierintehdent appreciate 
ihe. power of data and use it|in fotnnMng^plicy . *piey have been better 
4ble to justify policj? initd^tives^ pndj^'ave been | more assertive in 
claiy.ng initiatives to'chang^and i|i^^by4 the eduqitional system. In- 
formation from evaluations '<tf the.'Mi<^nfgan educa^^nal system has been 
used to form policies, and tn Curii J Jche policies have^ Influenced the 
direction of. the evaluatipttjl J > \ : " 

' i I ' ' ' 

The proactive, State*. Sdard. and|State Supwintetident in hiichigan / 

used evaluation activities/ and dat^ to establishyal state presence in, 

education during the L970'^. 'The'^denor of the^mbs was an "outcomes" . 

orientation ai^ a promoCiott of ejquijty and equallt;y for all children < , 

in education. The state a^epted 2J responsibility . for setting standards^ 

/for measuring impact^ and for asafi^ing schools tov^ard improv^ent* 

^This was a middte rpad between the uolicies in other states oA settidg 

statewide" grad^iatlott stand^fd^T^s^P ^ competency tests, and Iea^^ng 

standard setting £0|rpletel}| to laca^. initiati\^e. 
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THE INTERACTION BETWEEN POLICY AND EVALUATION 
IN STATE DEPARTMENTS OF EDUCATION 



James H. Gold 
GOLD Resources, Inc. 
Madison, Wisconsin 



INTRODUCTION 
0 

Although educational poli^cy^and evaluation have been a part 

of the American education s^ystem from its inception, * the content , 

form and relationship between them h^ve changed throughout the 

vears. Early school policy was gove^rned by concentration on the 

3 R*s and educa,tiohal evaluati^on was based on. the effectiveness 

of the individual teacher. In contrast, schools today have 

expanded programs far beyond the 3.R's t5i an effort to provide a 

, . ^ ' ' ' 

more comprenensi ve education to greater numbers of students. The 

public'has charged schools with the responsibiUty'^of addressing, 

and ameliorating /. social problems. Accompanying this expansior\ 

of responsibility is an increase in puDlic ^dollars frorrf local, 

-^-a^e-- f ^derarJr- s^taif-ee^-. As -cc^s ^s- have rirs^n- and restjurtres-^- . 

become less available, funding agencies such as -private . 

foundations, state education agencies, local education agencies^ 

and the federal^ government have placed increased importance on 

promoting and funding educational activities that encourage 

desirable' student behavior. Thus, program eva^luation has become 

an increasingly important part*- of general school operations. 

Specifically, ^the passage of the Elemei^tary and Secondary 

Education Act of 1963^ propelled the evaluation niovetoent by 

requiring^evaluat'ion of State Educational Agency (SEA) and Local 

Education&l Agency (LEA) programs supported by federal funds . 

This concept^ has grown horizontally ta other federal programs and 

vertically to state .and lotal ^)tograins/ Subsequently, evaluation 



and policymaking have become an integral part o'f management 
concepts such as PPBS', managemejit by objectives, ^nd the more 
generic concept of accountability. Although variations exist in 
evaluation ph il-osophy, and application, a common theme is that 
. feedback about behavioral changes resulting from program 
evaluations should be the basis .for both policy and operation 
decision making at the program level. 

The binding of program evaluation to policymaking is based on 
tne notion tnat tne scientific process which encompasses the 
evaluatien process would solve the problems of educating American 
youth just as the same processes were able'to put an Ajnerican^ on 
the moon in the ''O's. Thus, the expectations were rarsed'that a 
one-qg-one relationship could be developed between ev^iiXation and 
po*iicymaking . 

Today, oyer a decade later, evaluators, educators ^ ' 
educational decision makers and other interested parties ^re 
faced^with the reality that the one-to-one anticipated ^- 
relationship has'simply r^ot .evolved between policy-and. 
evaluation. Too often evaluation results are-'ignored in ligris.. 
politi-cal expediency. Consequently, we are^aced wnth the 
problem of improving utilization of evaluation information, in^ 
policy development. This -paper will detail some of the issues 
and present .how one state, Wisconsin, is structured to r^iata 
evaluation to policymaking. 



of 



I 



ERIC 



n 



CONTEXT AND "MODEL FOR EVALU'ATION ANP-' POLICY 

1. Contextual Factors 

— , ^ 

Prior to» discussing zhe relationship between educa- 
tional policy-making arfd ^valuation, it is essential to ' ' 
'understand ,some , of the import aat contextual factors withi 
which policy-making and evaluation operate in education* 
These factors are not new but are a-, reminder that the^ 
educational enterprise is, dynairiic , conducted by humans 
who possess the strengths and frailties which detemine^ 
the outcome of all human endeavqrs. Systems are made up« 
of people who should be accountable for results gather 
'than for the failure of the "system" itself. Thus, the 
following contextual factolrs are presented as a framework 

policy-making and Sva 
Contextual Factor 1 . 



within which most policy-m.akmg and Svaluation: zake place 



Decisions in education m^ade by influencing 

&-^<(ho have the '^ifnal decision-making author- 
ity inves^:^d__Lj;v-"t1iem by state coh^sti tutions .and ' 



^ Contextual Factor 2 
Ed^ation is a political prot:ess which is str^gly 
influenced by individuals and groups who are af- 
fected by the deci.sions\ Their vested interest 
may/ conflict with the welfare of* others. 

Contextual Factor 3 . * 
Education is an e'nterpz>ise in which peoplq with 
diverse values must agree to live with a single 
set of policies and operations within an edut:a- 
tional system, . • 

> Contextual Factor 4 

' = i 

Education is not an exacting science in which 
success or failure can 'be precisely predicted 
for any particular policy or program. Thus, a 
single ^est policy or program may fail to emerge.. 

Contextual Fartor 5 
Translation of pi^onrams and operations from policy 
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may bear little resemblence to the intent of 
the original policy* 

Contextual Factor 6 . 
Both individuals and groups are often wing 
for limited resources, which often leads to 
conflict and, competition rather uhan coQpera- 
tion. 

, Contextual Factor 7 

Evaluation conclusions are often contradicted 
and refuted by those desiring other outcomes-; 
They ma'y reinterpret data or pr^esenr "^contradic- 
tory data which supports^ their vie^/HDoint. 
_ Because of these contextual factors we must begin '^^h 
the assumption that educational decision-making regarding 
policy^ programs and evaluation is not always ration^, 
but is often based pn influence and compromise. This poli- 
tical decision-making orocess is inefficient but successful 
m a muxti-culttrral demacratic society in which public edu- 
cation is laid open, dissected, studied and gradually re- 
constituted in an evolutionary^ process . 

Within this context we can begin to fit together the 
'dynamics of policy and evaluation, realizing each state 



is somewhat different in terms >af the power structure, 
values, and tradi.tions which influence the poiicy/evalua-^ 
^tion relationship.' The" importance of this paper is for 
the reader to gain insight into. the dynamics of other states 
36 that he may better utilize evaluation results in his 
policy development process! 

2 . A General' Model 

, Educational polit:ies ar^ broad statements of intent 
which provide organizations with a basis for program de- 
sia^i'. and' implementation within a aiven State Education 
Agency (SEA) . They provide a description of agency direc- 
tj.on' to those outside the organization. Characteristically, 
polii:ies • lack quantification, specific behavioral descrip- 
tions and prOgram^ specifications. However, they usually do 
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reflect a desired standard." For instance*, a policy may 
be to '"promote equal educational opportunities for all 
children in the state/" This statement refiepts a stan- 
da:;d ijut does^ not indicate how it will be achieved or 
how one knows when it is achieved. These specifics 'are 
ac^mplished through the development of goals, objectives, 
programs fend program evaluation. 

The process Qf generating policies in an SEA is com- 
plex, and varies between SEA*s. However, it does appear 
possible to develop, a general model that reflects ti;e 
dynamics of 'decision-^making in mbst SEA'S. 

.Figure 1 (page 6) is such^ model ^^and is^ ^signed to, 
be flexible inorder'to accomcSda^e the variatio-ns which ^ 
occur m SEA's. The general ^flow of the model show^f^hat 
"data" raises policy issues , resulting in pdi^oy ac3op^ion, 
related programs and evaluation/ The evaluation^^su:^>s 
are then us^d in revising policy, bro^rains ^nd even ri^B? 
evaluation itself. ' - > * 

The model begins with the disclosure ^of "data", wnich 

strongly suggests, that either current policies be revised 
/ 

or new policies be developed. At the very minimum^ the 
"data" raises serious Questions ^ as to certain unmet needs 
'Which "imisT^^e-;aTidi!^ss'ea.^ "*^oa"t"a'*'""ih'This modeT^ " 
forms. First., people express^ thefir concerns based on their 
own experience^ a? parent^ , ed.ucato^, students, employers 
and taxpayers^. Although this data is neither systemati- 
cally' collected nor scientifically analysed it can have 
a powerful effect on educational policy, if a consensus 
of opinion is gained ' and the opinion^ are heard in a loud 
and clear ^fashi5n . This type of da^a can be more powerful 
in bringing about change than even the best evaluation 
studies . , I * 

The second type of '*data" is the moife systematic, 
consisting of test scores, surveys and research* Son^Q^ 
times these studies are carried out to reinforce or cha^i^ 
policy/while at other time's their influence on icy is 
accidental. . . ' 
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It is important to note that in most cases the two 
forms of "data'' are. used to' complement each other. For 
instance, many people expressed concern over students not 
learning the basid skills*. ^ This^ notion 'was then reinfor- 
ced by reports of declining test scores and other "hard" 
data. Thus, most state' departments have developed a 
stronger and more visible policy concerning basic skills. 

It is interesting to note that data has several 
points of entry into the policy development process. The 
chief State School Officer (CSSO) , governor, legislature, 
state board of education and SEA staff are all viable 
candidates for influencing policy. Depending on the per^ 
sonal policies of each and their relationship to e^ch 
other the entry points used' are based on receptivety to 
change and the power to change. -If policy change is de-- 
sired it is most ' important .to^ analyze the actors and sel- 
ect those who are receptive and willing to work for the 
desired change , ^ ^ 

Once £he da€a is revealed policy issues are' deter- 
mined by a number of different people. 'Professional \grdups, 
legislators, th^e governor,^ CSSO, and SEA staff offer var- 
ious policy alternatives. Each i^f the^e groups may prepare 
issue ^papers , appear at public hearings or attempt to 
persuade others on an individual basis through rational 
analysis', emotional appeal or political -compromise. The 
'end result is either no^ policy, or a policy which is agreed 
upon by either the SEA, legislature, governor, or any com- 
bination of the three. 

Policies are then translated into programs through 
the development of goals, objectives, activities and bud- 
gets. The legislature often goes beyond policy by deter- 
mining some program specifications.. Su<^h activity can 

* create conflict between *SEA and legislators. It is at 
this point that lobby ^groups /tirsp work to insure that the 
funds are allocated \r\a ^cti^Xiti^s are designed to meet 
the needs of their constituency. Thus,' these groups exert 

4 strong influence on both SEA Is and legislatures during this 
phase of the process. 
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state programs are given varying degrees' of autonomy in 
determining evaluation procedures. For some states pro- 
gram evaluations ajre required but the design* is left up 
to the SEA. Other ^programs have the evaluation specifi- 
cations spelled' ou^bither with general guidelines or 
specific ^activities . Regardless* of the form, -it appears 
that states' are increasingly required to evaluate educa- 
'tional programs. . * 

Evaluation results can influence virtually evety ^ 
phase of the model. Peoples' perceptions could be changed 
through the new information;, thus the "data" base ^changes . 
The evaluation could cause a re-evaluation of policy or 
raise new policy issue's. Certainly rhe management, or- 
ganization, goals, ob^ective^, activities and budget of 
a program could be affected. Whether an,y of these take 
place depends on the processes developed for handling data 
and the desires of those who control the data. 



/ 



ERJC ; - v« : 



FEDERAL. INFLUENCES ON EVALUATION POLICY 

^- Since the passage of the Elementary and Secondary 
Education Act of 1963, the federal government has bad 
greater influence on the policies ?and evaluations of state 
and local education agencies. 'The federal government of 
the 1960 *s began to intervene directly to overcome some 
of the large scale social problems of the decade- Educa- 
tion was among the many social programs affected. The 
major mechanism was the injection of massive funds to state 
and local education agencies for the purpose of designing 
and implementing programs that met the needs of society. 

In handing out money for specific programs the fed- 
eral government began to influencb state and local dis-^ 
trict policies. For instance, acceptance of Title I funds 
increased state and local district's committment to im- 
prove the education of the disadvantaged. Although most 
states h.ad some committment to this policy, the Title I 
funds inrcreased that committment and made it very visible. 
Likewise, the original Title III greatly influenced P^i~ 
cies for innovative programs and expanded the policy of 
publically documenting educational needs at a statewide 
level. Still other programs were responsible for'^imple- 
menting^ new management concepts and placeing.an emphasis 
on educational planrrlng on a statewide level. .At the SEA 
level many^ if not most, of the -current central planning 
units had their origins established' from ESEA fxind^ng. 
Thus, the federal government, through massive fganding y 
attached to specific programs, has had .a great impact on 
educatp.onal 'policies ^ the state and local levels. 

One of the key areas of influence is in evaluation 
of programs. A prime example is Title I, in which evalua- 
tions^haVe gone from being locally designed' to following^ t 
more rigorous federally mandated requirements. At the 
outset, local districts were* permitted great leeway in 
how their progrkm evaluations were designed and implemented 
The gpals and objectives were strongly encourage^ and the 
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eval^uation instruinents were ^eft up to \he local district. 
As a result, evaluc^idhss ra;?fged f rom 'excelle^^ to tot-aLly 
inadequate. As the^nconsistancy in quality became mof^ 
evident, the federal^guidelines for Title I eva-luatfbns 
were restricted to make them more consist'ant with good 
evaluation practice. . . ' 

In addition, congress began to seriously questio^' 
the effectiveness of Title I funds'. This questioning ' Ipd 
to char^ges. in evaluation requirements and subseauently to 
evaluation practices. When Taced w-ith measuring the 'im- 
pact of Title I, educators were unabl^to aggregate "Title 
I evaluation^ data from across the nation and gau^e fheir 
effectiveness. Instead, case studies and anecdotal data 
vere used to defend or arrack the missive expenditure of 
^ funds. Subsequently, congress mandated that a m.ethod be 
developed t^^ report on the impact of Title f to Congress. 

As a consequence Title I developed four models for 
evaluation which generated data: that cou^d be' aggregated 
at the state and federal levels. This. strategy limited 
the evaluation instruments that were required /LEA's 
could supplement), the sequence of evaluation events, and, 
to a certain degree, the content required* to be evaluated. 
In ess^ce, the Title I requirements limit the required 
evalu^ti^n strategie^s, curricular content, and test in- 
sfruments to those that Title I determine. 
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' LOCAL INFLUENCE ON POLICY DgVELOPMENT 

» 

Local individuals ahd groups can influence policy 
decision-making through several mechanisms. Lading de- 
finitive data on how effectively these mechanisms influence 
.policy, the reader must draw upon his own experiences to 
judge 'the value of each. \ 
1 • Individual Personal Contact 

In this case the individuals meuy call, write or meet 
with SEA management/ legi^^tors , or governor and express 
their' opinions . For people viewed by governmesjt as opin- 
ioh leaders, this type of contact is valuable. Otherwise, 
a large quantity of responses is needed to i^^ijji^nce policy 
decisions . 

2. Specific Issue Groups * 

'There are ad hoc groups which pool energy .and re- 
sources to change specific policies or procedures. Their 
interest is in a single^ tppic aiTd they iise personal ^on- 
tac1:s, -letter campaigns and media as their major mpdes of 
operation. 

3. Task Forces and Advisory Group's j 

Historically / the ^'governor ^ legislature and SEA 'have ^ 
established task forces to review specific problems and 
make recommendations for policy and programs. These task 
^rces are ysyally appointed by t-he governor and/or agency 
head, with the basisVof .selection not always clear. In 
most' cases , the local^ constituency is represented. How- 
ever, the individuals^ selected often hold views similar, 
or at least not incompatable with, the appointing author- 
ity. Thus many, but not all, task forces have a built-in 
bias • 

4 . Organizations , ^ 

The ^Ipcal constituency is usually well represented 
by variou^ professional organizations who have been in- 
creasingly involved in political lobbying'^ef forts to ^ ^ 
serve the needs of their members. Teachers, administra- 
tors and business officials all have their representatives 



who monitor ahd influence ^ educational policy and programs 
at the gtate level. In Wisconsin, some of these groups 
have formed an umbrella group with DPI for the purpose of 
discussing major policy and program considerations. Al- 
thqugh the group has no fojrmal authority, an overwhelming 
♦consensus on an Issue, policy or program would have a 
great^nf luence on those who make decisions. 

; In Wi-sconsip local influence has hati a general effept 
of maintaining loca'l\control where the federal or state 
laws have not compromised it. In evaluation, most ad- 
visory groups opt fo3* leaving tne design* and ilnplementa- 
tion up to LEA'S and ireauiring as little extra work as 
possible to accomplish evaluation requirer^enrs . This 
position is partly due to the issue of conVrol, but may 
very well reflect a feeling -on 6art of LEA's that 

evaluation data either is not, or cannot, be utilised 
enough to justify increa'sed demands on the time, energy 
and money of 'the loqal district staff. 
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EFFECT OF STATE POLICIES Pit EVALUATION 

State educational policies effect evalua^tion ,in three 
ways. First, the policies^ may determi-ne that no 'evalua- 
tion ^take place • This is usually accomplished by leaving 
the requirement for evaluation out^of Legislation and the 
budget.^ Thus, the programs^re implemented and a ge^neral 
fiscal accounting is done, but no performance evaluation 
takes place. 

Second, legislation and/or budget documentatipn may 
be very prescriptive in determining the evaluation policy 
and procedures. In such. cases the evaluation requirements 
are often spelled out in detail regarding; 1)" process-, 2) 
instruments, 3) time-lines, ^nd 4) reporting requirements. 
This situation pl^^&S^'evere limitations on the SEA,, but 
^increases the probability that the legislature will nave 
its evaluation policy carried out. 

Third, the legislature/budget requires, that the SEA 
evaluate specific programs that are being supported by 
state funds, * The directive to -evaluate is aften vaaue . 
sometimes ambiguous and always open to interpretation as 
to legislative intent. Procedurally, the C3S0 assigns 
the program to an individual to administer. The* nature 
of the program and evaluation will then be 'determined by 
the personality, politics, .professional persuasion, and. 
program priorities of the program director and hii/her 
superiors who have veto powers. * ^ 

Like^ individuals, organizational units .have t^heir own 

r 

personalities, politics and prog2;am' priorities. Thus', - 
central evaluation units tend to be interested more in 
performance based evalijations , /using Surveys and objec- " 
tive tests, \/hile instructional people often emphasize 
process evaluation, using interview techniques or other 
methods which place less dependence on student perfor- 
mance^ The nature of the evaluation, th^, will be estab-- 
lished by organizational " assignjnehts ^^since an ^overall 
agency evaluation policy is often absent. 
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INFL.UENCg OF EVALUATION ON POLICY 

Educational evaluation has the* potential for influenc- 
ing* four aspects of the educational enterprise*^ First is 
the establishment of programs based on evaluation of need. 
Once concerns are expressed as shown in the earlier deci- 
sion model, the pollection of systematic evaluation data 
may indicate the degree to^which the concern is real. This 
use of evaluation could set the course for the content, 
process, and extensiveness of SEA programs. The results 
of thfese evaluations can directly effect the amount of 
fiscal and human resources made available to address the 
concerns. 

Second, the management and operations of ongoing pro- 
grams may be modified as a result of formativ<e evaluation. 
These changes usually effect ti:*^ activities of staff and 
students, but. avoid any major changes in overall policy*, 

missions, goals -or objectivies. These changes are intended 

# 

to promote more effective and efficient attainment of the 
original goals and objectivies. 

A third aspect of evalucltion is program monitoring/ 
which, is related to management arid operations, but has the 
intent of insuring that the proposed program is the program 
being carried out. Program changes must be documented, 
verified and justified. 

The' forth area involves policy changes which deter- ^ 
mine the continuation of the program. Programs may be 
discarded because* the evaluation shows them to be ineffec- 
tive, ^inefficient or politi<^ally unnecessary, EvaluBtions 
may indicate that the goals and objectivies are both rea^ 
sonable and based on desired standards. However, the end 
results simply may liot meet the standards sufficiently to 
^ant cGrtitinuation of the project- It is also possible 
even if the 'goals and objectives are being met, the 
cost in dollars and human commitment . is too great for the 
outcome. Finally, when the evaluation results are placed 
in the larger context of^ total organj-zation programming, 




other priorities may supe,rcede the project as a result of 
chajiging needs and perceptions on the part of administra- 
tors and the publi<c. Thus, the evaluatidh may contribute ' 
to the expansion, maintenance or 'termination of existing 
programs . " . 

The final policy decision involves' the choosing of 
one prograrii approach over another, and the general signi- 
ficance of the evaluation results. As a result of a 
proaram evaluation an organization may choose to drop. one \ 
program and adopt and/or expand a program that the evalua- 
\ tion has shown to be more effective, efficient, and/or 
politically a<;:ceptable . 

In 'reality, evaluation studies have ^ rela^iively small 
Impact on policies in comparison to their impact on pro- 
gram operations. This situation exists because of the 
nature pf educational policy," the politics of the educa- 
tional ente^rprise, and the current state of the science 
of evaluation. 

As indicated earlier, educational policy is usually 

stated in broad and abstract terms which foster multiple 

interpretations of goals, objectives, programs and evalua- 

,tion . Since most evaluations are designed to measure 

program objectives and activities, it is Tittle wonder 

that policy is barely touched. In addition, most policies 

are robust enough to withstand significant program changes 

♦ * 
without requiring policy changes'.' 

Statewide educational policy is usually^ very appeal-- 
ing to the public and appears to be in the same' unassaLl-. 
cible category as "Chevrolet, apple pie, arjd the American 
flag," Within this context much pul^ic policy is tradi- 
tional and insulated from rapid and extensive change. 

^'his stability is due to the balance ^of power between 
traditionalists, moderates and libera^ls who influence ^and 

^ make 'policy decisions. 

This balance of power also explains, in part, the 
impbsing role that the, federal, government and courts have, 
played in bringing about both policy and program changes 



at the state and local level. During 'the last 25 years 
the government and courts have been liberal regarding 
social policy and programs, and have created policy and 

* , program changes at the state and local level by legal 
mandates Bnd the infusion of large grants for educational 
progra^ms. For example-, 'Statewide policies . concerning 
equal educational opppirtunity / school desegregation, 
school finarrce, and library building programs^ were all 
changed dramatically bea^use of federal intervention* 
Even curriculum po^icies^ and programs have been influenced 
dramatically by the creation of ^the National Defense Edu- 
cation Act (NDEA) , Title III, and Title IV* It is impor- 
tant to note that the federal activity came about as a 
result of public concern and a nationa^l' feelin^^ that th^ 
policies and programs were necessary for the 'public good, 
not because a comprehensive evaluation concluded th$ ^ 
changes were imperative. Society expressed concerns and 
the federal government and cqurts responded by establish- 
ing programs and policies that addressed them* • • 

Evaluation has had a less than desirable effect on 
policy ' because the results are of ten inconclusive o^- con- 

. trary"to previous studies* The technology of evaluation 

is not perfect and contains the bias of both the evaluator 

» 

. and the program staff which focuses on specific aspects 
of the program while minimizing aijd/or ignoring others. 
Even comprehensive evaluations have errors of both content 
^and design '^hich allow 'opponents to criticize the evalua-' 
tion and discredit the results on techical grounds. 

Similarly, 'education evaluation has not produced 
insightful, permanent, and significant discoveries which 
would -revolutitDnize the enterprise as is true in other 
fields. Evaluation discoveries tiave no analogies 'to X ^ 
^rays, penicillin, or the electric light bulb. Instead, 
)^ucatipn has a series of fads such as MBO's, programmed 
instruction, and the open^ school, which loose their lus- 
ter after a relatively short period of time or are re- - 
futed-by contrary research within a decade* As a result, 



educators and evaluators have failed to create the public^ . 
trust which other fields ^ave developed through finding . 
permanent and ef^ctive ways to address public concerns. 

^ Finally, th^ ^traditional view of organisations having 
specific 'missions towards which all their human and fiscal 
resources are devofed is no longer appropriate. Factors 
such as limited resources , "controversial policy issues, 
influence of special interest groups, public politics, 
organization politics ^nd the openess of the democratic 
process have led to decisions 'that are less than optimal 
in regard to the organization's mission, but more practi- 
cal in th^t they try to satisfy all the variables listed 
above so that programs and policies can be implemented. 
Thus, one should be little surprised when paarents and 
others express , dismay over thei,r perception that the 
children seem to have been lost in the decision-making 
process while the survival of individuals -and 'organifa- 
tions appear to have been optimized. Although evalua- 
tors should be concerned about students , in reality evalua- 
tion results may be neither necessary nor effective when 
dne or more of the above factors is given higher or' even 
exclusive priority i-n decision-making. 
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STJIUCTURE AND FUNCTIONS OF POLICY AND EVALUATION 
WITHIN THE ^WISCONSIN DEPART MENT OF PUBLIC INSTRUCTION 

' -s ■ ■ 

1. Department Organization 

The Wiscojisin Department of Public Instruction is 
unique in that the state superintendent is both' the major 
policy maker and administrator for education in the -state. 
The CSSO is a co^isti tutional officer e]^ected in, a popular 
non-partisan election ev^ry four years. There' is *no 
state board of education or any other\tate structure 
which supercedes the policy-making authority of the office. 
Thus, the CSSO is . accountable only to the public "every 
four years. 

Implementation of policies and operations are accom- 
plished in two ways. First, Vftsconsin is under a biennial 
budget system with a budget review occuring on the. off . 
year. The budget is a mix of ^i^cal and program elements . 
However, the budget has been. used increasingly by state 
agencies, legislature and governor for developing or 
changing policy* . ^ 

The DPI creates a budget which is submitted to the 

, Department of Administration for review after which the 
governor makes his-^rSUcnnrunendations fee the legislature. 

*Both budgets are reviewed and ^modi'fied by committee^ and^ 
finally adopted as a total state budget. The governor 

o - 

does have .line veto powers^'which can be overturned by a 
2/3 vote of the ^legislature^. This procesr^ generally fol- 
lows the model described in the* previous section -and is 
greatly ilifluenced by various special interest groups. 

Organization of DPI, shown in figure 2 '(page 19) con- 
sists of the *stata, .superintendent, an appointed deputy » 
and five appointed assistant superintendents who serve at 
the pleasure of *the CSSO. The department' is divided into 
five major divisions, including Financial Aides, HahdicaE|ped 
Children Instruction, Libtary, and Management, Planning ^ 
and ^ Federal Services. ^ ^ 
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' rnVISrOHAL STRUCfUKE OF WISCONSIN STATE DEFARTMLNT OF i>UUt.fC INSTRUCTION 
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An Administrative Council made up of the CSS^, deputy 
CSSO, and five assistant superintendents review' ma jor ^ 
policy changes, (in addition the CSSO confers with ^indi- 
vidu afassi stant s^^perintendents , program staff, and 
numerous task forces, and advisory groups for direction^ 
concerning policy, and operations. However, final decisions 
regarding policy and . implejnentation are, the sole respon- 
sibility of the CSSO . ' ^ ' 

2 . Evaluation Structure and Funqtlojis 

Hpu^d within the Divis'ion for Management, Informa- 
- tion and Federal Services is the Bureau for Evaluation, 
Planning,' Information and^ Research . This is considered 
the central evaluation unit of DPI and is divided into 
three units, two' of which are involved directed ia P^^^" , 
gram evaluation. The* unit entitled Educational Planning, 
Evaluation and Research consists of six persons who have 
varied responsibilities. Some design and implement eval- 
uations for other PPI programs such as special e*ducation( 
and nut^ritional education. Others review and monitor \ 
local evaluations for Title IVC projects^ Some have^ been 
j^volved in a statewide nepds assessment which had poten- 
tial for statewide policy development. For the most part, 
the group is "on loan" to provide service^* to others who 
lack 4taff to fulfill their evaluation needs. This group 
also^advdses local districts .on the design and implemen-- 
tation 04. local evaluation programs. 

The second unit is the State Assessment .Program which 
conducts an annual statewide assessment of pupil perfor- 
mance. This unit has a staff of six. In addition to state- 
wide testing, this unit has been instrumental in (developing 
a local option •liesting program ^nd is in the beginning 
'Stages of developing an item bank t^o be used by LEA's. 

3 . Evaluation ?1ethodoloqies 

The basic methodology used in evaluation is that of 
establishing outcome objectives -and measuring the, attain- 
•ment of those objectives for each program evaluation. In' - 
addition, some process evaluation may take place to insure 



that the program proposed was in fact the program eValu 
ted. Achievement tests, interviews and qiiestronnaires 
aye all been emplo.yed as data-gathering tools-. 
Problems and Constraints 

Lack of clarity concerning 
what clients want from evaluation 
Many clients come to evaluation without %uch ^ 
• knowledge of what questions they want the 

evaluation to address. Subsequently, a gre^t ^ 
deal of time must be spent on clarifying goals 
' and objectives. To some clients this is both 
a tedious and often threatening task. . 

Attitude 

Some clients are' afraid of evaluators because 
they are intimidated by them and/or t4el they 
are being personally evaluated. Such feelings 
^ainder the evalifation process in -that people 
either become resistant to the process or *agree 
to things they later reject under the pretense' 
that they did not understand them. , ^ 

Minimal Effort ' ^ 
Clients are often being forced into evalua-tidh 
to hold on to their funding source, and^ there- 
fore, desire to do the minimum to meet the re- 
quirements. Tjiey are just going through the 
motions . 

Communications 
Evaluator and clients often fail to communicate 
ideas and assumptions, thus a common understand- 
ing does not exist. Such communication gaps ar6 
caused by both language differences and varying 
degrees of receptiyity to ideas arid viewpoints 
on the part of both the evaluator and client. 

Use of Evaluation Results 
1 - 

After the ^time and money are invested, after 
fulfilling an obligation to evaluate ,^ client^ ^ 
do not utilize results adequately. There is 



resistance to pte-pl^nning^^)ie use of results 
and a propensity for a "wait-and-see" attitude, * 
However, most evaluators do believe that results 
are used in a less systematic and less visible 
fashion. 

Difference in Philosophy 
Differences in evaluation philosophy exist among 
.educators, especially in terms of the "hardness" 
^or "softness" of data required for adequate . eval- 
uation. In the Wisconsin DPI, as in most agen- 
cies, there are those who place greater emphasis 
on procf^ss rather than outcomes. Thus, conflict 
arises as to how much reliance should be placed 
on studen't performance data vs , other types of 
information. If ^n evaluator whg is a strong 
believer in perfonnartce data is "loaned" to a 
program whose personnel tends to believe more 
in process or otTier data, the evaluator is C9n- 
strained.in applying his prpfession as he/she 
believes is appropriate, • 
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In additjion, the CSSO directed that public involvement be 
significantly increas^ed in all phases of the assessment. 
This change resulted in the assessment of practical skills 
and knowledge in addition to the purely academic objectives 
the assessment had previously focused on. 

The changes in policy and procedure were consistanr 
with the general policy of the CSSO regarding local con- 
trol. The assessment was to be developed for, ^nd b^^^, the 
public with technical assistance provided to local di"^- 
"tricts on a 'volunteer basis^ The assessment design wa-s 
a • "safe"^ compromise which showed DPI was concerned about 
accountability and yet was not demanding enough ro pose 
a" direct^, threat to any particular- special interest group. 
In summary, the assessment was a reactive program designed > 
to quell concerns over student performance. Its purpose 
was to provide public^infonnation and hot* to comprehen- 
sively ie^valuate specific educational programs in Wiscon- 
sin • , 

> 

Although the yearly results made interesting news- 
paper coverage, it is difficult to identify if the assess- 
ment has been instriomental in changing policy. Mo new 
^programs have been initiated and no n^w fun<Js have been ^ 
generated by the assessment results. There'* is scant evi- 
dence that the educat^LrOn of Wisconsin children has improved, 
or that any improvement, which may hav^ occurred, would be* 
due to the assessment. On the positive side, th^re is 
evidence that local districts using the local option pro- 
gram, have .used it for prog^m improvement. This effort 
4iS encouraging, since districts are attempting^ to find 
ways to m£ti^^,valuation figure more prominently in policy- 
making! \ * 
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PUPIL AgSESSMENT; A CASE. .HISTOR^ 

In 1971 the Wisconsin legislature with' the" support 
of the DPI enactec? 3.115.28(10), which mandatecf the*de- 
partment establish a pupil assessment program yi thin very 
broad guidelines: 4 ^ 

i 

Develop an educational program t'o measure ob- 
jectively the adequacy and efficiency of e^fi^u- 
cational programs offered by public school^ in 
this st^te . . . Assessment shall be iindertaken 
at several grade levels on a uniform' statewide 
basis. (3.115.28(10) 

Unlike other states, this legislation passeq with a 



minimum of de±)ate and little organized qppositio|^. It 
received neither wide press- coverage nor special^ interest 
group attention. Most importantly, the legislat\:y:e did 
nqi provide any state funds for initiating the program, 
which may account for the lack of interest among ^Law-mak- 
ers and educators regarding the passage of the bill. 
'Thus, the DPI was mandated to provide/ a -program a£ pupil 
assessment, but was given little legislative guid'^nce, 
interest or fiisiding. \ 

' Fortunately, the' state superintendent was committed 
to thk concept of accountability, and therefore allocated 
discretionary- federal funds for starting ^ the program with 
the intent that the state would eventually take ov^Tr its 
support. This became a reaLity in the following biennium, 
when the state took^ full support »of the program. The state 
increased allocation for the program each biennium since. 

From 1971-75 the assessment program developed a set ^ 
of \^goals , for education, and implemented assessments in 
reading, mal&h, science and social studies. The Eleven 
Goals for Education 'w^re intended to be the basis of the 
assessment and were to provide direction for education in 
Wisconsin Public SbhoCls. The test instruments were all 
objectively referenced, and 'put together by Wisconsin 
educators, or selected from Nktional Assessment of Educa- 
tion Progress (NAEP) . They were administered to a random 
sample of public school students based on a^ two-st^ge ran- 
dom sample design. 

^ ^ ^os 1 1 ^ 



During the first four years, several events did and 
did not 'take place* No definitive purpose for the assess- 
ment was established or documented, ^There^was little con- 
sideration given to the nature and type of data to be - ^ 
collected, ^There was little consideration of:persons 
responsible for content and technical quality pf the in- 
struments. No one was designated responsible for the con- 
tent of the reports-. Thus, an internal tug-of-war began 
between assessment personnel, the specialist, and manage- 
ment of ocher divisions. As a result, until 1976, piroducts 
were da^ed unadequate by t^e Assessment Director. Many 
policy-makers, outside of the DPI, questioned the value • 
of assessment data. 

At the end of the 1975 Assessment, an internal evalua- 
tion of the assessmenc program was done by^the assessment 
staff and a Technical Advisory Committee, The evaluation 
concluded; the sampling procedures were excellent, the 
logistical systems for administering the program were ex- 
cellent, the test instruments needed som^ refinement, and 
the prcJgram did not seem to meet the jxeed for student in- - 
formation. Thus, an assessment program had been created 
that operated well, Jput did not satisfy needs of educators, 
law-makers, or the general biti^enry of Wisconsin, 

As a j:onsequence , the CSSO directed the assessment 
staff to acco^nplished the following: 

1, 'Prj^yide the 'citizenry of Wisconsin with a statewide^ 

profile of the -quality of education' as reflected in 
students % ability to demonstrate expected knowledge, 
ski^lls, and attitudes, 

2, ' , Provide state officials with student performance in- \ 

formation for use in educational policy development / 
and/or communicating with bonst^ituents , 

3, ' Provi.de state officials and the general citizenry 

with a profile of Wisconsin pupils* performance as 
.compared to a national average, 

4, Provide school districts with the opportunity for' 
'self-evaluation, using the methodology and products 

Cf the Wisconsin Pupil Assessment Pirogram, 
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HEURISTICS FOR INTEGRATING POLICY AND EV^IJJATION 



'For the^ pr.upose of clarity it is necessary to draw 
a distinction between local project evaluations with lim- " 
ited local implications, and statewide evaluations which 
have potential implications for the entire state education 
system. Policy questions are more easily addressed in 
projects. The appropriateness of a decision can be judged 
against the cofic.erns and needs^ of a "^relatively small- group 
of people ^instead of judging whether a paticular policy 
or program is appropriate for 4 36 LEA*s who have both com- 
i mon and unique needs. Consequently, policy may be more 
controversial at the statewide level than in a 'single LEA» 

Tke sheer size of a statewide evai^uation and deci- 
sion-making process opens complicated political channels' 
which- ara, often difficult to control. The power structure 
may shift .with the policy issue in statewide processes, 
whereas, at" the local level the power structure appears 
to be more stable. 

Statewide evaluations are usually carried out by^ SEA 

staf€, while local project evaluations itiay involve SEA' 

staf,f, local staff or outsijJe consultants. Since state- 
* ♦ 

^ wide Evaluations require ^an internali, eval^ator, his/h^^ 
role may be different than an outsider's. The internal 
evaluator begins with a particular statids in tKe organ- 
izationand is, in all likelihood, less prone to, deviate 
^ from the organizationNal nonps of ccmranunication , a^tt^tudes, 
and innovation. This individual ^is already part of the 
organi/feationa^ strucrture and, in all probability, h^s - 
been pigeon-holed into specific political and philos(^hi- 
cal categories, making it- Iqss liJcely that ihe 02:gani/2a- 
' V ' tii^al staff j^ill view the evaluator as unbiased, Th 
SEA evalu|?f:or must then make a conscious dec:(.^on to 
either facilitate the evaluation process by providing ^ 
l&echnical expertise, or take ar^ active role in the^ poli'- 
tics of the situation, thereby influencing the design, 
impJi,ementation and utilization of the evaluation. 
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Project Evaluation **» 

A. ''Make ^sure- y.aur credibility with the organization 
is established by providing documentation <*f work his 
tbry and ueger'ences fropi other evaluation projects " 
yod have conduct^Fd. Submission of *an evaluation re- 
port which -you have completed would also be valuable. 

B. Determine at an early state both the formal and 
informal structure for decision-making in the organi- 
zation, and deal' with the appropriate decision-makers 
regarding the evaluation design and implementation. 

C. All communications should be dane at a language 
and technioal level appropriate for the audience. 

Do not use the average level but the' lowest level in 
the audience ,so that you wiTl^ communicate effectively 
witn all people i-nvolved in the process. 

D. Be prepared to deal with ^people^gp^^a simplistic 
level. Assiyne responsibility* for increasing the 
audience * s . knowledge of evaluation techniques and ^of 
howi results can be utilized. 

E. Clarify the roles and responAbilities <5f the 
evaluator, staff and administr^ation • 

F. Do not go into an evaluation situation with pre- 
c'onceived notions of what the' evaluation should do or 
how the evaluation will be conducted. Be sensitive 
tc local needs vs.^ textbook .models', , In -the final 
analysis, the be'st evaluation design wi^'l be. useless 
unless it^' meets local needs. 

G. Do not take sides on local ' political issues, but^ 
act as a conciliator in bring about oompromise. Make 
suggast^ions an^i^rela^ relevant research, but do not 
impose your' viewpoint. ' ^ - * " 

H. Identify the purpose of the ^valuat:ion and^ the 
specific question^ that need to* b^ answered j This 
is essential since most clients do not understand 
what an evaluation' can- do or what they want. Be 
honest about what questions can and cannot be ade- 
quately addressed, ai\d explain why. State how much 
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st^ff time and what resources are required for answer- 
inq^ questions. Do not let the client-expect more than 
the evaluation or evaluator can deliver*. ' Document the 
purpose of the evaluation and the specific questions 
to be addressed and have both the client and evaluator 
stgn it% -^^ < 

I. Develop the evaluation process plan. Include acti- 
vities, responsibilities and timelines.. This may be 
accomplished through developing options which the client 
'decides from, based on your analysis. This decision 
should be a team process so that? the client will feel 
ownership and try to tailor the evaluation to the 
local situation. 

J. Develop an analysis 'plan that specifies how the * 
data will be translated into answers to the evaluation 
questions. ^Avoid statistical jargon. Present ^he 
plan so that everyone involved will understand the 
^process and the final outcome. 

K., Develop an interpretation and utilization plan 
that delineates who, how, and when the results will . 
be interpreted. Thi^ step is critical in utilization, 
since, mpst clients are, inclined to let the. results 
^ deteifmine the utilization of *data. By defining how 
specific desirable and undesirable outcomes will ^f- 
^%fect p61icy and operations, the cliej^ wijLl probably 
^be more committed to following through with specific 
. actions . ' , ' 

L^. Develop a dissemination plan which targets evalua-- 
tion results and 'recommendations to audiences in a- form - 
thatlthey will redd and follow through on. Produce- - 
technical and summary reports which convey the same 
information "^in different forms.' In some cases dis-^. 
semi?)ation ipay utilize alternative media such as trans- 
parencies, television or slide/tape presentations, 
M. Keep ke^ decision-makers informed as^ to the* pro- 
gress ji>f the evaluations and any unusual findings. 
Do not drop surprises on the client to which he/she 
is not ready to respond • 



I 



i09 



Make sure the role and responsibility ..of the evalua- 
tor is clear in terms of information release Do not 
release information without the clients approval. Re- 
quests far information should be directed to the client, 
2 • Statewide Evaluation and Policy Interaction 

Most statewide evcrfeiabions are conducted by the SEA 
staff, thus,^there is little client/evaluator conflict. 
However, the evaluator must interact with other government 
and public entities, findings the most resis/tance in intra- 
agency dealings. Thus, the heuristics presented below 
assume evaluation and policy interaction are internal agency 
activities that may involve extra-agency politics, 

X. It is essential that the evaluator' have the full 
confidence of the agency '-s top management, 'Since 
statewi de d ecisions are made by the CSSO and/or state 
board, it is imperative that the evaluator build a ^ 
strong track record with them. He/she must pb:5duce 
evaluations that fit their expectations and are 
politically astute,, flexible and technically c6m- 
petent, ^ ' - 

B. The evaluator ^must build ^a positive relationship) 
with ^those staff members who may.be affected by eval- 
uation outcomes. Agencies are generally resistant 
to change. Individuals may be directly threatened 
by evaluation , and may do everything in their power 
to overtly and covertly impede the evaluation effort, 
(2. Evaluators should build relations with 4-nfliien- 
cial groups outside the agency and educate them as 
* to how evaluation results can be used in making de- 
cisions, ^ These influencial people include budget 
analysts , legislative^ staff , professional organiza- 
tional staff and media people, Evaluators can edu- 
cate such ]|)eople by holding conferences and work- 
shops for them, pr by meeting with them individually ' 
to inform them of evaluation progress . Under no 
circumstances should such meetings take place if 
they conflict wifth an agency policy or rule, Like- 
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wise, the evaluator should not promote ideas which 
are contrary to agency- policies , rules or ^ regulations . 
The intent is to educate and build confidence in^he 
evaluation proces^s. 

D. The purpose and objectives of any statewide eval- 
uation should be clearly delineated ^ afid approved by 
the CSSO and the evaluator. The evaluator should 
offer alternatives and recommendations ^as 'to what 
questions the evaluation may address, as well as, an 
analysis of the policy and operation implication of 
eagh. 

E. ' An evaluation plan should be dev^^loped that in- 
cludes activities, responsibilities, and timelines. 
This plan is then signed-off by* the CSSO and manage- 
ment staff who are involved. The signing of the plan' 
^represents a commitment to implement itf Mana^ge-' 
ment must insure that staff members carry out the 
plan even*if it means reduction* in other staff acti- 
vities. 

F. / An analysis plan should be developed to specify 
ho\ the data will be translated into a format that 
will answer the questions addressed in the evaluation. 
This\plan provides a mechanism for . communicating what 
will and will not be done with the data- 

G. A utilization plan should be developed to indi- 
cate how the evaluation results will be used. Posi- 
tive and negative results should be analyzed and 
potential actions described. Such a plan represents 
a public commitment to use data in specific ways.' 
It may prevent the collection of extraneous informa- 
tion that is costly and inconvenient. 

h. Provide a dissemination plan that directs eval- ' 
uation results and recommendations to target audien- 
ces in a form that they will read and*follow through 
on. Produce technical and summary reports which 
convdy the same information in different forms. In 
some cases, dissemination may utilize • alternative 



media such- as transparencies, television or slide/tape 
presentations • ^ ^ 

I, Have evaluation results reviewed. by professional 
and lay people who have different points of view. 
Provide mechanisms for each interaction with the two 
objectives of, 1) having people with diverse view- ' 
points gain a better understanding of each other and 
the issues, and, 2) obtainiiig new perspectives on 
the issues. Such an open dialogue promotes common 
understanding and increased support for subsequest . 
actions • • ^4 

Although the heuristics described above may aid util-* 
izing evaluation results in policy-making, •-the dey to 
success is in the persqAalities and politics of inqivi- 
dual dec.ision-makers . It is obvious that some decision- 
makers 'have the confidence and ability to attempt revolu- 
tionary changes while others are satisfied to let the 
"system" evolve at its own pace. Regardless of any par- 
ticular situation, the evaluator must understand that 
evaluation results will always be used in^^a political 
context. Unless attitutes change, most systems will de- 
cide to provide sufficient, not optimal resources to ^n 
organization ih helping it attain its goals. Credible 
evaluation, then, 'is essential for verifying needs and 
effectiveness of programs within the broader context of 
decision-making.* 
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EVALUATION AND DECISIOW MAKING IN WASHINGTON STATE 

Alfred Rasp, Jr. 
Washi-ngton State Department of. Education 



BACKGROUND 

On January 11/ i973, a program evaluation" section was officially 
established ij^ the office of the Washington State Superintendent 



of 'Public Instruction; and for the first time, at least in modern 

histotyr^^an emphasis was placed on the measurement of program 

impact. This does not ,mean that previous superintendents lacked 

♦ ^ 

interest in the success of programs, but it did express a new 
concern, for generating evaluative data as a basis for policy 
making. This paper will attempt to describe both the 
organization changes that have taken place a'nd the interface 
between evaluation arid decision making in Washington state. 

To make sure there are no misunderstandings about intent ^ 
names or geography, three ground rules will be established. 
First, this description of events will be neither an -expose' of 
agency practices n'or a positive self-serving statement lauding 
the efforts pf ttje evaluation section. In the words of Howard 
Cosell, the. goal is to "teli' it like it is." Second, in addition, 
to the use of the standard educational acronymes such as LEA") 
SEA, and USOE, the Washington State Super iritende'nt of Public 
Instruction will simply be abbreviated to SPI in the name of 
economy. Tbifd, to avoid misunderstanding whenever the word 
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Washington appears singularly, it will. mean "State of" not 
"D.C." People in Washington state ]ust' prefer it that wayi 

Dr, 5rank B. Brouillet was elected SPI in the fall of 1972 
ahd officially launched his administra^tion January 11 , .1973^^^^is 
prof essio,nal' career riepresents an interesting blend of fe^dljcation 
and politics.' He has professional experience from both the 
school and college levels. He is a former teacher / counselor, 
coach af^ administrator. He has degrees in economics and educa- 
tion and an earned doctorate. Perhaps most unique in this 
background blend — he served 16 consecutive years in the 
Washington House of Representatives -and consistently provided ^ 
legislative leadership in educational affairs. 

This combination ',of experiences hassled Superintendent 
Brouillet to a three-part educational philosophy. He professes a 
firm belief in the importance a local control, a commitment to 
providing the resources necessary for a quality education and' a 
dedication td^he- basic tenents of educational accountability. 

It is the third element that is of special significance to 
<5his discussion of evaluation and decision making. Being an 
insider to the wo'rking of the legis*lature , Brpuillet knew long 
before being sworn in as SPI that perhaps the only way to expand 
the amount of state resources for education, and at the same time 
protect and jStr engthen local control, required close attention to 
accountability. He knew that maintaining or increasing the 
fmanciar support for programs in, existence and initiating new 
programs ^depended in large part on. providing the legislature 
assurance of the following: first, that the program is 
necessar/"that the need really exists; and Second, that the 
impact of the program can be measured, anb, o'f course, tfhat the | 
results are ^sitdva^ The methodology of evaluation plays a 
The -key Question, however, is not whether one 



central role 
alternative or 



treatment is more efficient or effective than 
another./ but in a more basic sense/ does the alternative selected 

a difference? Is there an impact? Assuring need arid 
effectiveness become prime concerns ^r evaluation in the 
political accountability system. The influence of an elected 
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superintendent with an educational and legislative perspective 
clearly makes an impact on evaluation practices. 

\ 

ESTABLISHING AN Evl^UATION SECTION 



The rhetoric of the campaign trail/ increasing programs, and 
protecting local conCfta^by establishing need and measuring 
i^ipact became criteria for establishing a new section within the 
office of the Superintendent of Public Instruction* This program 
evaluation section gave concrete, visual proof that the new 
superintendent meant what he said — there was clearly a pj^e to 
point to on the organizational chart. (An organizational chart 
appears at the end of this chapter.) m 

It should be nqfed that in many respects the ^gvelopment of 
the evaluation section was a process of putting "new wine in o!^ 
bottles."" Because at the same time the super intenden^^ook 
office, the legislature in the name of efficiency placed a limit 
on the staff size of state agencies, and no trained evaluators 
jwere hired. The evaluation section was fprmed with personnel on 
hand, only the responsibilities were new. ' * 

As time pasfed, the deputy superintendent was instrumental in 

shaping the section into an effective work unit and ej^panding the 

» 

emphasis on program evaluation. He recognized early that in the 
name of objectivity program managers should not evaluate theij 
own programs and .that outside contractors could not interact 
favorably with the legislature. With his leadership several 

programs were designated as priorities and small amounts of their 

I 

administrative funds used to establish project employment 
. positions in the evaluation section and to hire staff to carry on 
" the evaluation activities. This move gained both objectivity and 
credibility/ as well as the required evaluation data. 

The title 'Of the section has changed during the years to 
Reflect new emphases a^id to better reassure the ]^gislature and 
the public that th^ accountability charge is being carried out* 
In the beginning, the name Program Evaluation seemed to be the 
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answer. It soon became apparent that not having a "research" 
descriptor appear in the organizatiorxal roster was causing the 

agency to miss important contacts. Thus the section title was ^ y 

expanded to Program Evaluation and Research, 

By the mid-seventies, however, evaluation in the SEA setting 

had generally subsumed research actlvities^and, with the advent. - ^ 

of "golden fleece" awards and other indicators of *the public's 

low esteem for educational research, the title was changed to 

Testing and Evaluation. This choice dropped "research" and added 

emphasis to the more popular notion of "testing." At a time when 

legislative debate'on questions of testing was long and loud, the 

reference to testing in the. section title reflected the S?I*s 

intent to meet issues head-on, . " - ^ 

A broadening of the accountability concept took place in the 

mid-seventies when the Seattle School District successfully sued 

the state for not meeting the constitutionally mandated duties 

"to make ample provision for the education of all children, , ?and 

provide for a general and uniform system of public schools," ^ 

his legal battle led directly to the passing of the 'Basic* 

Education Act (BEA) in 1977 and to the need lEor "evaluating" LEA ^ 

compliance with the provisions of the law. At the same time, .the 

State Board of Education renewed its interest in expanding the 

concept of school, accreditation to 'focus on faculty 

self-evaluation, that is, on improvement through evaluation. How^ 

better could th^ S?I meet these , accountability challenges than by 

adding t^^ responsibilities to the Testing and Evaluation section ' ^ 

and changing the titile to Testing, Evaluation and 

Accountability? In 1980 the routine compliance checking activity 

of the Basic E^5ucation Act will move out of the evaluatiqn sphere 

/» - 

to a more appropriate, long-range setting. Perhaps the sect>^o^ 
title will stabilize ^s Testing and Evaluation, * 

This discussion of names may sound superficial, but it is 
important to note* that the activities of 3 the section have' always ^ 
received the necessary financial support. Hopefully, the major 
reason for this fortunate circumstance is that the section staff 
has discharged its assigned responsibilities With professional 
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competence. The changing of .titles, however, does reflect an 
attempt to match the "mood of the times," certainly that of t^e 
legislature, and to build confidence in the SPI's intent to 
establish program need and to measure imp4ct as decision making 
base* 

CURRENT RESPONSIBILITIES IN TESTING, EVALUATION 
AND ACCOUNTABILITY 

t 

Testing ^ 

Currently, the ma^or testit^gL ^respo p^-r^ility is' to carry out' 
the mandates of the state testing law passed in 1976. This law, 
tjtled Student Achievement Surveys and Tests, requires states 
testing and reporting at three grade levels. The SPI must 
annually aaminister a standardized achievement test in the basic 
skills of reading, language arts and mathematics to all fourth 
grade students. The results of the testing, along with the 
relationship of achievement to appropriate input variables, are 
to be reported to the legislature, LEAs, and s^bsequently to the 
parents of ^children tested so that parents can compare the 
achievement levels of their childTren w4th others in the district, 
state and nation. In. grades eight and eleven, samples of 
studefjts' siJf f ioj^ntly large for generalizing to the entire state, 
approximately 2,000 at each grade level, must be tested in. 
reading, language arts and mathematicsr and the results reported 
to the legislature at least once every four years.' The law also 
encourage-s local school districts to conduct diagnostic testing 
in grade two but does not assign that r,esponsibilty to the SPI. 

The mam intentions of the legislature in passing the laW are 
clear. There was first an interest in ascertaining ^the impact of 
basic skills Jlistruction. This was typified by the guestibnsj 
What are the achievement' levels of Washington students? How does 
Hajshingtan^ertorm an cQ c o mpar-Q-w4-feh— feh e - na t iu ii dl? D o ' d ie dS " ~ot—^ — 
weakness requiring special attention exist? A second purpose, led 
to the display of district summaries of fourth ^rade test 



results. The interest being two-fold — to spotlight high 
achieving districts in order to learn from their- success arid to 
isolate low achieving districts for special assistance* A third , 
purpose was to provide parents ^nd the public information about 
the' impact of 'Schooling, th^t is^ ,to encourage educators to more 
fully share information related to program outcome^. 

^ The law is implemented through heavy reliance on contracted 
services. To accomplish major tasks such as the printing and 
scoring of tests, logistical services and analysis, requests for 
proposals are prepared and sent to interested bidders. The 
technical proposals submitted are reviewed by outside panels of 
experts working independently. The recommendations of the 
technical review panels are supplemented by the S?I staff 
analysis of bid amounts; the superintendent giakes the final 
decisions, and contracts are written with successful bidders. In 
Washington, contractus for $2,500 or more require that a 
«ompetitive bif^ding process be used. Single source cootracts for 
larger amounts must be justified ^nd defendedv* In the case of 
contracting with other state agencies, ' for example, universities, 
educational service districts and LEAs, waiving the competitive 
bidding proCess is not difficult; Jiowever, when agencies other ^ 
than those of the state lire involved, great care is taken to 
explicitly follow the rul^s. , 

/ 

Since the total professional staff responsible for the 
testing activity is 'less than one full-time equivalent, 
-contracted services are necessary and play a ci^ucial* role. The 
typical pattern is^o^e in wbicb large contracts for specialized 
services are awarded 6n the basis of technical merit and 
competitive bid. The assistance of additional personnel is 
gained through contracts with the other state agencies or school 
districts. Specific tasks are completed occasionally through the 
use of single^ source personnel service contracts under the $2^500 
amount. Developing work plans and time schedules, preparing 
requests for proposals, reviewing bids, writing a'nd"managing 
contracts are necessary skills for administering the Washington , 
testing program. 




^' The testing results are reported in several forms and through 

several cnannels. In the case of grade four individuals, 

classi^m,' schodl, an<3 district level reports, including summary 

data and* item analyses are delivered to the LEAs as soon as 

possible after the October testing. In December, the state's 

performance is publicly released to the LEAs and media. By the 

end of\pebruary the State General Report and District Level 

Summaries is-disseminated to the legislature, -LEAs and media. 
^ 

With the sample studies at grades eight and eleven, there is less 
information to report. Since a sample is used, no classroom, 
school or district reports exist^. Wber\^ -possible, individual 
student results are returned to the schools, but the reporting 
consists primarily of a news release of the state's results 
folJLowed by a general report sent to the legislature, ' distr ict 
superintendents, principals of schools with the grades tested and 
the media. 

An additional thrust of the testing program aims at helping 
personnel in local districts to improve their skills in 
selecting, administering, interpreting and reporting test 
results. This effort usually takes the form of workslio^ 
conducted throughout the state. The first' series, timed before 
the October testing of* fourth grade students, focuses primarily 
on test administration. A second series, conducted after tjfe 
state's fourtb grade test results have .been returned to the 
districts, emphasizes interpretation, reporting, a^nd use of test 
results for instructional improvement. 

Evaluation 

Major evaluation efforts revolv.e around the evaluation of 
selected, priority program. These are the programs in which the 
SPI has a special interest because they involve large sums of 
njoney and/or are compensatory or categorical in nature and/or are 
pal^iJfericaily -sensitive. 

For the past two years, evaluation priority has been placed 



on six programs: Title I, Title I-Migrant, federal programs for 
t 
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the handicapped, the Washington Urban, Rural, Racial Disadvantaged 
program, educational clinics(^or ' dropouts, and the Title IV, Part 
B learning resources program. Although these are all designated 
as important, the evaluation responsibilities Vary from program ^ 
to program, with greatest efforts in Title I and Migrant, and ' 
least in the areas of Title IV, Part B, 

In both Title I and Migrant, full annual evaluation reports 
are prepared for USOE, These reports are based on' the ^ 1 
computerized aggregation of data from application's, monitoring 
forms, interim reports, anq year-end reports, as' well as fiscal 
files and program offices files. The annual evaluation reports 
describe how the program resources were used, what outcomes 
^suited, what trends developed, and what special problem areas 
existed. The annual reports also show the extent to which the 
S^ate plan goals and objectives were met, 

.In addition to the preparation of the evaluation report, the 
computer ^ata files are summarized and printed to provide 
perxodxcal management info^nation for the program staffs. During 
tbe course of the year, the e'valuators also assist in training 
LEA personnel to use program forms and procedures. 

There are two points of emphasis related to tbe evaluation of 
federally sponsored activities for handicappea students. For 
several years, the main responsibility was for evaluating the 
special state pro3ects provided by federal discretionary funds. 
This was accomplished through year-end report data and on-site 
reviews. More recently, with the impact of ?.L, 94-142 and the 
mandated individualized educational programs for handicapped 
students, the evaluation staff has been worki^ primarily to 
assist in the develdpment of a computer p'rocessing system for 
management information, including an emphasis on organizing 
monitoring and evaluating data. 

The Washington Urban, Rural, Racial, Disadvantaged program 
was expanded in 1979 to include Remediation Assistance {RAP) . 
Whereas the regular URRD program has provided money for a wide 
range of crisis oriented pro3ects for tbe past decade, the RAP 
addition is strictly a compensatory program modeled sufficiently 
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aft^ Title I to qualify Washington for the Title I incentive 
grants when the federal funding becomes available. The 
evaluatiop section involvemeht with URRD takes many- forms, 
including^;, the review of the evaluation plans specified in the 
granl^^ applications, onsite project evaluations,^ the computer 
aggregation of compliance monitoring' data, the follow-up study of 
student^ served, and a computer summary of application data. In 
the case of the new RAP component, assistance has been given in 
the .development of program guidelines for LEAs and in the 
preparation of the reporting documents. The year-end evaluation 
activity will include the preparation of a statement on 
achievement gain in the style of Title I. \ 

Evaluation assistance to the manager of the Title IV, Part B 
learning resources program typically has taken thtee forms. The . 
application and financial data are stored, aggregated ^nd 
tabulated by computer for both interim management information and 
year-end reporting purposes* The results of LEA compliance 
monitoring by learning resources program staff are entered into 
the computer and aggregated. Three to five case studies 
involving onsite re^^ews of LEA activity were prepared by the 
evaluation staff during each of the past three years. At ,the end 
of the year, computer printouts of updated program information 
and moxiitoring reports, along with draft copies of the case 
studies*, are delivered to the program manager, who is responsible 
for preparing the annual report for USOE. . - 

For the past two years, th§ State Board of Education has been' 
required by legislation to certify education clinics organized to 
provide programs-for school dropouts, and the SPI has been 
required manage the funding process and evaluate the programs. 
Because of the special legislative interest, the activities are 
politically sensitive beyond the small amount of money involved* 
The law |tself ca|.ls for the* evaluation of superior performance 
based on educational gain as related to thq difficulty of 
educating the students and efficiency in terms of per pupil 
expenditures.' The demands for evaluative precision outstrip tbe - 
current state of the art. An achievement and superior 
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per formang^"^ report is prepared annually based on data aggregated 
from maivroual stuaent recora forms th$t are submitted by tl^e 
clinics for each st'udent entering and exiting the program. From 
this inforjnation , a description of each clinic is prepared* 
showing^ a difficylty to educate factor, an achievement factor, 
and an efficiency factor. 

Accountability and Other Responsibilities 

Since the Basic Education Act went into effect in September 
of 1978, school districts must be judged in compliance, or have 
certain regulations waived, by the State Board of Education 
before the SPI can distribute the funds provided by the 
legislature to t1?em. With 100 percent of the funds for the^a^ic 
program moving to districts through these channels, the 
determination of district compliance is of^cru^ial importance. 
For the past two years, this responsibility has been fulfilled by 
the Testing, ^Evaluation and Accountability section. Fo*rms are 
developed and distributed; reports reviewed. Recommendations 
based on the district input are presented to the-= State* Board of 
Education for* actio/i. As the board judges districts to be in 
compliance, the SPI*s Division/of Financial Services manages the 
apportionment of funds. Approximately one trillion dollars flow 
to Washington's 300 school districts through this process each 
year. 

^The school accreditation programs are also administered by 
the Testing, Evaluation and AccountabJ^ty sectipn. In 
Washington, two accrediting programs interface. The .State' Board 
of Education by law must provide an accreditation process to any 
school that applies. The program is voluntary, not a basis for 
"funding available to all schools. Although the State Board has 
accredited secondary schools for years, in 1979 the l^w was 
amended to add elementary schools. The adcredi*tation "^fyrogram is 
currently in the developmental, field test stage-, and the. goal is 
to have a process operational by September, 1980. The second 
accreditation program is that of the Northwest Association of 
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Schools and Colleges. * This program is managed by the section 
director who serves as the chairman for the Washington Committee 
of the Commission on Schools and as a Tru,stee of the 
Associatfion. Th^ Northwest secpndary school accreditation 
process toas oppra^ted in Washington since 1917 and currently 
involves approximately 180 member schools. Management has been a 
responsibility of- the director since he joined the SPI staff in 
1970 and a responsibility of testing and evaluation ^nce the 
section was formed. Educational im^Jrovement is the goal of both 
programs. The central elements of each revolve around 
determining that 4:he resources required for a quality educational 
program ajre present and conducting an indepth self-study with 
external verif ic^ltion. The state board activity as part of the 
SEA activities may move to another section in 1980, 

Section personnel afre responsible for a number of other 
activities — some of which are closely related to testing and 
evaluation and others only by a great stretch of the 
imagination. For example, liaison is provided with a number of 
Organizations, including: American Educational Research 
Association, Wa§h*ington Educational Research Association', ^ 
Northwest Evaluation Association, National Assessment of 
Educational progres?. Northwest Directors of Assessment, 
Coimnittee on Evaluation and Information Systems, Region X-Title I' 
Technic^ Assistance Center, Northwest Association of Schools and 
Coilegelf Nationil Study of School Evaluation, Association of 
Washington School Principals, piemen ta^ry School Principals 
Association of Washington, W^ghington Junior High/Middle School 
Principals Association^ Washington Association of Secondary 
School Srincipal^/ High School-College Relations Committee/ 
Washington Pre**CQllege Testing Program^ and the Washington 
Alter nat'^ye'-LeSrrfing Association. 

Questions dealing with correspondence schools and high school 
graduation requirements are also answered by section personnel. 
In addition, the section is the clearin^ghouse for many research 
activities in areas related to testing and evaluation. A current 
example 'is the **High^ School and .Beyond" study being conducted by 
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the National Opinion Research Center (^r the National Center for 



Northwest Reading Consortium, a four-state Research and 
Development Ut?ilization Program established by the National 
Institute of Education," was coordinated by the section. Section 
staff also provide .technical' assistance in planning, testing, 
evaluation and research as requested within the state agency anp 
outside. * ^ 



During fiscal year 1980, several staffing patterns will be 
used to provide the human resources necessary to complete the 
assigned responsibilities. At thds time there are eleven people 
regularly on the section's payroll. Seven of these are 



The seven professional staff members have all been teachers, 
but their backgrounds vary greatly* Five^ave earned doctorates, 
and. they bring great diversity to the section because each 
studied a different specialty. JFor example, one received the 
degree in curriculum and instruction, another in educa'tional 
psychology , a third in counseling and guidance, a fourth in 
reading, and the last in the administration of higher education. 
Of the seven, two are former principals, one a school counselor, 
one a school psychologist, one a reading "specialist, and one a 

former state education association president* 

• * 

/ A*s a result of experience and graduate study, all have 
backgrounds in educational research, but none have extensive ^ 
formal training in evaluation. This is not to say that there is 
a lack of expertness. Since the section was launched in 1973, 
steps have been taken to develop the required skills. Through 
individual initiative, section, staff development activities and 
on-the-job training, the staff has gained a high professional 
level of competence. In the areas of large scale assessment, 
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professional educators and four are secretaries. 
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program evaluation and the use of ?he computer to facilitate the 
aggregation of evaluation data, the pjofessional strengtn of the 
section is noteworthy, it should also be noted that the section 
has earned a good reputation — a high level of credibility even 
though it frequently deals with tough topics that are not always 
viewed favorably by LEAs or by others in the SEA. 

The seven professional staff members represent two hiring 
^^patterns and three funding sources. Six of the seven are regular 
civil servants. The seventh is hired on the basis of special 
project neeo, and the employment must be renewed and approved at 
the beginning of each project year, depending on the availability 
of funding. The section budget is based on three sources of 
funds: state money provided by the legislature for the testing 
progam and for general SPI activities, such as administering the 
Basic Education Act and State Board of Education's accreditation 
program, federal dollars for state leadership in education and 
small amounts from th4 administrative money of Title I, federal 
handicapped, state compensatory and federal learning resources. 

.The regular secretarial staff consists' o'f four people. There 
are two full-time secretaries, one part-time secretary working on 
program evaluation {^reports, and one part-time Secretary assisting 
in the computer processing of data. As overload situations 
arise, temporary help is added as required with a minimum of 
bureaucratic strain. 

Eleven people cannot attain all of the objectives flowing 
from the many assigned responsibilities, but with the size 
limitations imposed by the legislature, there a^e not staff years 
, available for hiring additional permanent personnel. Steps are 
taken to augment the staff through the use of personal service 
.contracts. In some instances, the contracts call for another 
agency taj)rovide personnel who will work under the direction of 
the sect ion • In ofher' cases, the accomplishment of spec'^ic 
ta^s forms the prime objective of 'the contract. Occasionally 
when the need for assistance is short in duration or the specific 
task is small in ^scope, a personal service contract may be 
negotiated directly with an individual. 

-' ' ' • i^i 
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Using contract has both advantages and/^isadvaritages^. 

Certainly control over the size pf the permanent staff is 

^ maintained, and there is an effic^i«nt flexibility for peak load 

staffing. However, the negotiating, writing, defending, and 

managing of contracts is time-consuming and frequently calls for 

. efforts over and above the time normally spent on the 

superivision of personnel. There is also a potential problem in 

th'e lack of staff continuity and commitment to long-range goals. 

Because of the different types of contractual arrangements used, 

f 

it is difficult to estimate the number of fu»ll-time .equivalent 
stvaff members that serve the section during any given year, 

RELATIONSHIPS WITHIN THE AGENCY 

There are a number of relationships with the SEA that help to 
define the roles, the responsibilities, and, in a- sense , the 
location of the evaluatin unit. 

Success in fulfilling the evaluation responsibilities depends 
on close and positive working relationships with SPI program 
managers. The evaluation staff and program staff negotiate a 
work plan specifying the activities, timelines and staff 
responsi^jJUrties that will guide the evaluation effort throughout 
the year, ^he c^^cle of involvement typically begins witli a 
review of a program's state plan. Placing emphasis in two areas, 
the objecfe*^es are analyzed to ensure that they reflect the major 
intended outcomes of the program, and the evaluation plan itself 
is elaborated and brought up-to-date. Application and reporting 
forms are 'examined to make sure that they will provide the 
information required and provide it in a cohdition compatible 
with computer data processing. With the assistance .of the 
computer, information is aggregated and reported to managers on 
the predetermined schedule. The outline for the final report is 
""^^^ .discussed with the program staff, and draft copies reviewed 
before final printing. This review is conducted to provide 
program staff an opportunity to point out possible data errors 
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and to provide a first-hand knowledge of tpe contents before the 
report is disseminated. 

In order to promote objectivity, the evaluation and program 
activities are clearly separated by housing each in different ' ^ 
divisions of tne LEA, None of the programs for which an annual 
evaluation repp^rt is prepared are located in the Division of 
Instructional and Professional Services — the home of the 
evaluation unit. This separation solves two problems. Since the 
-program staff is-rigot evaluating itself, there is an appearance of 
greater objectivity. At the same time, since the evaluation is 
being conducted within the agency, there are evaluators who do 
know program strengths and weakness — needs and outcomes — and who 
ban provide credible data and testimony to legislative bodies and 
funding sources. This is an attribute or advantage that outside 
contractors typically do not have in Washington, * 

The relationship between evaluating and agency policy niakin<^ 
fls less clear than that at the program level. The assistant 
superintendent heading that division which operates the programs 
approves the evaluation work plan, and the director of testing 
and evaluation briefly discusses the planned activities with the 
deputy superintendent and the SPI, The3e interactions, however, 
are frequently routine, resulting in statements , like, "Sounds/ 
good, let's do it," The attitude is no| negative or 
disinterested, rather it reflects confidence in the negotiated 
evaluation P^n and the evaluation procedures being used^ Simply 
stated,^the position of the SPI policymakers seems to be: the 
30b is getting done, there have been no great problems, funding 
sources are happy en^gh — why make changes? Why disrupt the ^ 
process? . • 

^ I 

There are additional relationships, A computer playing a ^ 
central role in the processing of evaluation data generates 
anot^r set of interactions; and good working relationships 
between ''man" and machine are crucial to the smooth 
implementation of the evaluation process. In 1976 a 
minl-computei^ was acquired to help solve the problems brought 
about by an abundai^ce of work and a shortage of staff. Although 
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the progranuninq><rVe become more sophisticated and 

the amound^^^^l^a processea^.has grown, the basic human tasks 

• hav.e remained the same. Efforts go into the streamlining ^of \ 

application and reporting forms to make them more efficient for 

\ , ' * 

the entry of 4ata into the computer and the aggregation of ^ * 

essential decision making information. The items<>on the forms 

are coded/ if necessary, and entered ii^ the machdne^' The 

reports of LEAs are printed out and returned to them for 

-correction. The data are aggregated to m^t^ with the requests 

of progoram m^agement, and the final updated computer files a^e 

used as the-basis for preparing the annual evaluatiorj report. 

Learning to work with the computer has be^n difficult f-br some 

program managers, but the system is expanding and providing a 

broader range Of evaluation services each year. 
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Definition and Pur^s 
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In a recent roundtable discussion, members of the evaluation 
staff were , asked to define "evaluation," and in every ins 
• the responses represented a variation on two themes. -Firs 
evaluation was described as an objective process of ' collecting 
and organizing data, of judging impact and ascertaining value. 
Second, everyone agreed that the definition was not 'complete 
without a statement of purpose; for example, recognizing that 
evaluation is conducted to assist managers in making decisions. 

Although thi^ definition is broad enough to encompass the 
generally used strategies, two additional elemenl^s were described 
as necessary- in a dtjccessful evaluation. These risixolve 'first 
"around^ the clear deline'ation of what data are required; that is, 
i^hat questions are to.be answered and, second, around the 
effective displaying and reporting of eva'luat^e information. 
Following this definition, the SPI ev^luaqSE effort 
< includes: deciding what data are needed t(5v ansf^r^the key 
evaluation questions, collecting and organizing pe data, ' 
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providing the necessary comparisons and judgijents, and reporting 

^the information in a- way useful for decision making* 

Ideally, the purpose of Evaluation is to provide sufficient 

information about program alternatives so that managers can 

easily see the comprative value and make d'ecision^ that promote. 

effectiveness and efficiency. The ideal situation generates 

informationj which reinforces the need for the program treatment 

and shows its impact* The evaluation activities ideally would 

follow a linear sequence in which n'eeds were determined, 

ob3ectives set, programs implemented, outcomes measured and * 

' — 
information required to guide the next program cycle. In real 

life, however, the process is often abridged and seldom on the 

time schedule implied by the planning model. In most cases, the 

data generated are more important/as formative information for 

program managers than as summary data for high-le^fel 

policymakers, and certainly the data -are' more descriptive than 

judgmental. 

Data collection is the easiest phase of evaluation r but it, 

must be clearly established what decisions are 'going to be made, 

what data ai-e needed, and when the analysis and report must be 

r^ady. In_g$neralr data are collected about the program targets, 

a^ut the actual "performance" or outcomes and about the 

resources ^used. More specifically, several questions must be 

answered to guide data collection in the Washington evaluation 

♦ 

process. What needs are addressed by the program? What 
objectives are included in the state plan*^ What implementation 
strategies are suggested? What outcomes are expected? What 
resources are provided? What groups are involved? Who is 
'Served? What are the treatments? HoyT ar^ the resources used? 
What are the program outcomes? \ What comparisons are appropriate? 

Two additional questions influence the Effort. Who is 
interested in the outcomes of the evaluation?' Who ought to be 
interested? .The answers are important for reporting purposes, 
but they also help to solve delineation problems in the"^' future. 
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Five Uses of Data 

1. Annual reports are prepared to meet the requirements of 
state and federal funding sources. These reports generally 
attempt to provide the answers to the previously mentioned- 
guestions guiding the' data collection. 

2. The data are also used to assist program managers in 
be'coming more effective and efficient. Management memos are 
prepared as. part of the annual reporting J)rocess, but the 
audience is SPI's program managers and policymakers, not the* 
funding sources. The goal is to help the state to better meet 
its obligations^ through improved practices and quality control. 
The content of a memo may vary from the conpents related to the 
need for improved office practices (for example, better written 
documentation of program changes) , to the highlighting of 
objectives not met, or questionable fiscal practices. Although. 
_th_e^_rj;igrain managers ax^^oJL_alwfl^?=; pXeag^H with- the cohtent<^~ - 

this use of evaluation data is viewed as a constructive practice. 

3. The deputy and ^he super mtendent vrely on the evaluation 
section to keep fhem informed of any special circumstance that_ 
could ultimately require their attention. A third major use of 
the evaluation data is to make sure that policy makers are not 

"suddenly "confronted with an unpleasant surprise. They want^ to 
know in advance, for example^ about the unpredicted concerns of 
special interest groups, anticipated major management problems, 
and possible audit exceptions. 

4. An abundance of descriptive data is available in the 
computer f iles ,to provide information for decision making — a 
fourth major use. The files include, for example, numbers 
served, money spent*, time in programs, delivery modes, . staff ing* 
patterns, f>arent advisory committee activities, and program 
outcome^. The data are arranged by districts, by programs, and 
by funding, source.- The informtion is especially useful to 
program managers because it provides an up-to-date reflection on 
how resoi^rces are being used, that is, who is being served, in 
what ways, and at what costs. Through the aggregation 6f project 
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monitoring input/ the managers can also review which projects are 
, out of compliance and which rul^s or regulations are causing 
problems In the field, ^ 

5. Policy makers working at a different level of decision 
making abstraction use the ^ata in two special ways. First, by 
reviewing the information available, they can keep the impact 
^measures promised to funding sources ab part of the 
accountability process more reasonable and within the range of 
possibility. This sensitivity is critical if credibility is to 
be established and maintained, i Second, the data are used to 
support dfecisions previously made. Since ofteo the timing of 
^ll^licy jnaking and the collection and analysis of evaluation data 
cannot happen in the preferred sequence, the required data are^ 
estimated on the basis of past experience and.upd/ted whei^ the ^ 
actual data become ' available, hopefully to confirm the decision. 




^ Evaluation and Policy Making • > ^ * 

^ THe term "policy maker" refers to the actions of the* 
tQp-lelvei SPI managers as they develop the budget, the 
legislative thrust, and provide direction for the agency and the 
^ overall operation of the Washington common school system^ * This 
management level consists of, tbe superintendent and his 

in^ltrative 'staff-, the. deputy, and the five assistant 
superintendents who head the five agency divisions, Tfte SPI is 
ao elected official, and the members of this policy group serve 
at bis pleasure- and are exempt from the state civil service rules, 

If^^B specific policy questions arise. Section directors, who- 
are tenured state employees and provide prc5fegsional^ontinuitv. 
are frequently invfted to join in the policy-making ^Kceedinga. 
For example, the director of testing and evaluation provides 
Significant input into policy <Jecisions- regarding the state level 
activities in that; area* Describing this interaction as 
providing' inpat,** however, does not reflect the fuJLJ^ range of 
he dynamics. Although the process is not formalized in SPl' 



o;perating, procedures, t^re is an act;ive two-way exchange, , The 
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director does participate in policy making related to testing and 
evaluation, but perhaps mote importantly S?I policy' makers rely 
on the section director t^ keep abreast of educationa'l, 
legislative, and executive ac^tivities/ both state and federal, 
and to t^ke the initiative irt providing them necessary 
it^formation. The director, in a sense, is asked to be an 
advocate ^of sound professional practice and to also be able to 
discut^'s the impact that alternative decisions would' have on 
various components of the educational^^p^amofilTy", with interests 
in testing and evaluation. In addition, it should be noted that 
just as open discussion a»nd full input are expected before a 
decision is made, once it is made, everyone is 'expected to fully 
support its implemen tart ion ► 

The development of the state budget provides two examples of 
evaluation's involvement in policY^making. Along with the amoun^t 



" of funds j;^uest^d, each item in the budiget must be defendfe^ with 
statements demonstrating the existence of need and describing the 
measures that will be used* to show impact. Th^e state testing 
program is a specific budget entry, so there is a direct policy 
making interaction regarding the activities planned for the 
biennium covered by the budget, the amount of money that will be 
required for ''implementation, and how the need and impact will be 
described in relation to the mandates of the state testing law. 
Since there is a tendency in organizations supported *by budget 
allocations for sections or programs to" attempt to show their 
value in terms of the amount of resources they command, the 
budgeting process frequently leads to a compromise. The demand 
for resources is greater than the supply, and a compromise 
between what is desirable and what is necessary &y law and 
required for state leadership results. The series of * 
negotiations is active and positive, and reflects both the 
superintendent's and legislature's priorities. 

The testing and evaluation section is additionally involved 

^ in the budget making process t)irough a technical assistance 
role. To maintain a solid reputation with the Governor's Office 
of Fiscal Management and the legislature, the needs, impact 
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measures; ar;d' outcome data specified in the budget must.be 
deliverable. Program managers 'and even the SPI executive staff 
are tempted occasionally to promise data that cannot be 
obtained. They are especially tempted tp promise the measurement 
and reporting of achievement gain as an indicator of impact 
because this is the '*hard" data that funding sources prefer. ' The 
problem is that in most instances it is not possible to deliver 
Over the short run, the use of h;gh sounding impact measures may 
bring fundid'g; in the long-run, the loss of credibility outweighs 
the temporary advantage and causes major problems. The section 
is called on to help guard against this happening. 

■ The development of the Washington Remediation Assistance 
Program provides another example of participation in policy - 
making* In an effort to gain more resources- for the support of 
tlie schools, the super int^dent , with the advice of his 
administrative staff, decided to seek legislation and funding to 
promote the remediation of basic skills deficiencies in the 
intermediate graaes. It was decided that results of the fourth 
grade testing prpgram as the most "believable" aata available 
would serve as the entitlement system for the allocation of the 
program's funds. The three years of- background data were used to 
establish the need and the distribution mechanism, and the 
experience with Title I evaluation and the USOE models became a 
key element in the evaluation design. This latter connection was 
important, since the state remediation program was being 
organized to qualify for the federal incentive grants under 
Title I. • 

As mentioned in the discussion of the URRD program, the 
section also assisted. in the development of RAP administrative 
guidelines and in the preparation of reporting forms, Tfie 
achievement or impact section of the annual report will be 
prepared by testing and evaluation personnel. 
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AREAS OF FRI^TRATION 
* t 

LiKe all evaluation units, the SPI^s unit has f^ced many 
frustrations. Some situations have been solved, others 
circujnvented/'but a number, however, remain consistently 
unresolved and irritating. The following 12 situations provide 
brief examples of problems, both philosophicaT and practical, 
which continue to frustrate members of the Washington evaluation 
staff. ^ 

1, A clear role differentiation between"* evaluation and 
✓^search has' not been established. As a result, there is a range 

of instructional activities which have never been verified as 
efficient or effective, and there are probably a number of 
teachers who work very hard to do things which may not promqte 
learning. Typi-caLly, the evaluator's prime role is to collect 
and organize data which describe program inputs and outcomes. In 
most instances, the researcher is interested in developing 
generalizations which explain or predict events. Neither 
researchers nor evaluators generally develop instructional 
materials or procedures, nor do they spend time checking to see 
if instructional methods are 'faulty or misused. There is a 
depressingly large twilight zone resulting from the unchecked 
assumption that the instructional methods used in a project 
accurately reflect the research finding^* and are^^T^eing used 
.appropriately. In fact, there is almost no research or * 
evaluation energy applied to the analysis of alternative 
intervention strategies in Washington. — 

2. The role of •'describing" in evaluation also requires 
clarification. The growing use of case study and ethnographical 
approaches sqggest several questions: Where does the description 
stop and evaluation begin? How can the comparative statefnents so 
frequently demanded of evaluation be made? The potential of 
using these data' ccj^lection techniques to assist in the program 
evaluation has not yet been fully realized, and the frustration 
of trying to harness the rich data into an evaluation statement 
with utility for an audience no on site persists. 
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Another frustration on the order of a "pet peeve'* is the 
inability or unwillingness of program managers to separate 
monitoring from evaluat?ion. Evaluation is clearly distinguished 
by an emphasis on program* impdct or outcomes. The on-site 
reviewing of projects to ascertain compliance with rules, 
regulations, stated objectives, ana negotiated budgets is an 
important management function, but it is not evaluation. To 
consider that the worth of a project can be determined by the* 
degree to which it is in compliance is misleading. The * 
responsibility for monitoring as a management function is moving 
more and more to the program staff, and the energy and resources 
of the evaluation section are focusing on the evaluation* 
questions of impact and efficiency. The movement is riot , 
complete; and to the extent that it is not^ the frAistration 
remains. . ^ 

4. A fourth disappointment stems from the fact that too many 
educators are willing to use evaluation as an end in itself and 
to limit progra© exaj^afies and alternatives to tbose aaaenabie to 
■good* eval^tion designs. Frequently, evaluators are blamed for^ 
causing this practice wben they probably spea^ most loudly 
against allowing the evaluatiojn to determine tl^ program 
parameters, Tbe situation has developed, or degenerated, in some 
instances, to the point where greater pride is taken in 
evafluatdj^n results than in the actual program outcomes, and this 
**Catch 22" scene appears to be growing. 

5. A great frustration also results from aggregating impact 
and related data from 300 Washington school distx^icts and 
watching important distinctions **wash out** in the averaging. 
There are successful projects and significant differences. 
However, implementing laws, for example, which call for the 
Correlation of "appropriate input variables** with the achievement 
of grade four students tends to often obscure the situation 
rather than clarify it. 

6. Using evaluation data inappropriately to respond to 
outside inquiry, or generalizing beyond the power of the- data, is 
a persistent frustration. An example, once. again from the fourth 
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grade testing, illustrates the problem. Frequently, letters come 

§ 

to the agency from people moving to Washington asking for help'' in, 
decic^ing where to locate. The usual response is to send a cc^y 
of the fourth grade assessment re^rt with district-by-district 
achievement results. This report shows fourth grade achievement • 
scores, district level per pupil ex^penditures (a .08 correlation 
wrth achievement in 1976) and an ave'rage fam?ly income figure 
cased on 1974 estimates"(a ,47 correlation with achievement in 
1976). The sending of^-the report by, implication suggests that 
contains reasonable data for deciding where to live. One might 
say that this information is better than nothing, but the 
frustration is — it i-s not sufficient information for fudging the 
quality of a school district. \ \ 

- * 7. Computer processing of evaluation data causes a 
frustration of major proportion, or perhaps more clearly, four 
frustrations. First/ the battle is still being waged against the 
iltentality that views the computer as something ma^ic, with the 
^^ility to aggregate errors into precision. Second, the 
time-consi^ing problems of moving data from report fibrms into the 
machine are not fully apprecj^ted by program managers, and the 
problems have not been solved completely. Third, working with a 
single, "one-'Owner** machine as compared to a service center, 
gives control, but a minor machine breakdown causes a major 
disruption of service. Fourth, the greatest frustration results 
when, even after lengthy plahning and negotiations, program 
managers demand answers to questions that are not compatible with 
the data collected, stored, and programmed. v 

8. The problem of gaining 'sufficient support service in the 
preparati6n of evaluation reports has not been solved. The 
desire^ editing, graphics, lay out, and printing skills are not 
readily available in the agency, and going outside for assistance 
is difficult because of the rules regarding the role of the state 
printer. 

9. The lack of congruence between thfe ev^L^uaXor^s ''logic" 
and political decision making "logic" is also a keen source of 

.frustration. The old and accepted political process of basing 
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decisions on pbwe^ relationships is effective in lining up 
support and getting certain jobs done The fact that power 
politics^ works, however ,^j:^s not detract from the' frustration of 
p/esenting objective and overwhelmingly persuasive testimony" 
based on - acj^iarate and logical evaluation data to a legislative 
j^ommittee and experiencing a contrary decision. 

10. Another large source of frustration arises from the 
inability to gain a clear delineation frpm the policy, makers 
regarding the decisions they will have to make, the i'hformation 
they will need; arict when it will be needed. A major reason 
the difficulty is that frequently * evaluation data only play a 
marginal role in policy making. In the name of accountability 
programs must be evaluated ^ and the evaluation report must 
indicate what needs are being addressed and what outcomes are 
being obtained. ^Too often this information is treated -as an end 
product, the report is maae and filed, but the information is not 
used for planning purposes. Evaluators by default often carry on 
the deli'neation activities vicariously and hope for aa accurate 
match with policy making needs. It is a difficult problem to 
solve f but repeatedly, clarification is lacking and the 
evaluation data ^re notion target and not useful for policy 
making. 

11. An ultimate set of frustrations revolves around time , 
dnis profe^lem is tfie nearly complete acceptance of the logical 
planning seqt}em:e which tends to mislead people into thinking 
that if they go through the steps they will automatically 
accomplish something. Ev^uators move back and forth through the 
sequence ia roany *dif ferent orders, and in iDOst instances probably 
start by trying to establish what people with a stake in the 
program -.would be/willing'to accept as ^evidence that it is working 

A second time consideration "than is frustrating could' be • 
labeled a continuity problem, and one specific example will 
elaborate the point. During a recent school year, 
repf esentatives of the evaluation sectfon joined with the 
university and educational service district staff to work 'closely 
with 12 small school districts on the Olymj^ia. Peninsula to help 
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in planning and conducting assessments and evaluations aimed at 
clarifying priorities and isblating problem areas in the 
curriculum. The effort was productive, and by the eild of the 
^scKdol year all involved had gain^ed professional satisfaction for 
a job well done. The teams were eager to start the next year's 
'round of activities. In September, however,, the excitement 
faded— half of , the dis,tricts had new superintendents and one 
district 'no longer existed. . » ' 

A third time element is the recognition that time, or more 
specifically, timeliness itself, is a critical variable in 
evaluation. Perhaps the supreme frustration is to conduct a 
sound evaluation, generate useful information, and deliver a well 
documented report—just after the crucial decision has been 
made. In the spring of'1974^ the legislature mandated that SPI 
conduct an evaluation consisting of pilot studies in LEA account- 
ability and a statewide assessment of basic sKills achievement. 
The law passed in April 1974, the funds became available July 1, 
and plans were developed in detail. Accountability projects were 
initiated involving a university^, an educational service district, 
and several LEAs in the- right geographic mix^ An achievement 
test was developed using items from the National Assessment of 
Educational Progress, a scientific sample of students was drawn, 
and the testing scheduled for April 1975. All e.fforts were aimed ' 
at making best use of time and dollars and having the study's 
report completed by June 1975. Of course, the legislature met in 
January 1975, asked for the information, wondered what was taking 
so long, and nearly passed bad legislation. 

12. A final frustration results from the fact that SP^I staff 
and staff assignments are constantly changing. For example, over 
the past six»years the Title I program staff has changed in some 
way every year, the management pattern for the migrant program 
has changed each of the past three years, and the assignments and 
personnel in the program for handicapped children have 
changed dramatically. Federal legislation and reporting 
requirements in all of these areas have also undergone major 
, transformation. In addition, reorganization within the agency 
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has become a way of life, and it seems as if the evaluation 
process alvAys involves new people in one phase or another* 
Petronius Arbiter captured the idea when he reported his ^ 
frustration in 60 AD: 

We trained hard, but it seemed that every time we were 
beginning to form up into teams we would be reorganized* 
I was to learn later in life h^at we tend to meet any new 
situation by reorganizing; ai>o a -wonderful method it can 
be for creating the illusion of progress while producing 
confusion, inefficiency, and demoralization. 

Of course, Washington evaluators do not have a corner on this 

frustration* «^ 

IN CONCLUSION 

During the 1970s, the emphasis on program evaluation gained 
widespread public popularity, as well as the strong.' support of 
executive policy makers and law making bodies* In fact, few" 
developments have made so thorough an intrusion into the 
operating practices of education. There has always been an 
emphasis on the precision measuring and accounting for resources, 
such as the number *of books in the library, pupil-teacher ratios, 
and the number of hot lunches served; but the stress placed on 
the evaluation of program results has come as an intricate part 
of the accountability movement. Other professions haye concerned 
tbeiftselves with various forms of input-output analysis for many 
years; the decade^^of the 70s, however, marks the general 
introduction of accountability and program evaluation into the 
educational setting* The concern for the analysis of resources 
used and results gained is real and growing* -Howevei:, the fact 
that there is a lot of **program evaluating" going on should not 
be- interp;reted as an indication of educational progress or 
confused with claims of program improvement. 

Evaluation has come \;nder criticism in recent times, 
especially from evaluators themselves. There has been a 
tendency f perhaps, to address too many of the tough problems 
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related to the conduct of evaluation intellectually rather than 
practically — some of the most reputable evalaators are spending 
more' time verbalizing about evaluation than practicing 
evaluation. One of the results is that new models, approaches, 
and strategies are developed and discussed, but the basic trouble 
spots plaguing "applied evaluation" remain unresolved. Crucial 
among these is the overestimation of the influende of evaluative 
information on management decisions and policy making. The 
intent of tl^iis paper was to provide background examples to use in 
thmjcing about evaluation issues and ways of improving practices. 

As stated earlier, few ideas h^ve spread more rapidly to 
permeate the field of education than the concepts of 
accountability and program evaluation. Making the movement pay 
off with imprj^ved practices and better education for those served 
is still the fchallenge. 
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THE INTERACTION BEPv^EEN 'POLICY AiND EVALUATION 
WITH^ THE SOUTH CAROLINA DEPARTMENT OF EDUCATION 

Paul D. Sandifer, Director 
Office Of Research 
^ South Carolina Department of Education 



INTRODUCTION . 
Purpose 



The purpose of this paper, as commissioned by the Research on 
EvaTuation- Program of the Northwest Regional Educational Laboratory, 
is to describe the interaction between policy and evaluation in the 
South Carol ina' Department of Education (SCDE) ... More specifically, 
how does policy influence evaluatiorr and how, if at all, does 
evaluation affect policy svi thin the agency? ' , ^ 

A literal interpretation of the purpose leads. on6 to attempt to 
view the SCDE In isolation from other state and federal agencies, as 
well as special interest groups^ Although such an approach would 
have the advantage of resulting in a much briefer paper, it would • 
ignore the considerable influence of other agencies and groups in 
shaping the policies- of the SCDE and in establishing external 
policies under which the agency must operate, Consequenftly, the 
focus of the paper is not limited to the pol icy/eval uation / 
interaction within the SCDE but also examines some of the ways in 
which various ageneies and special interest ^groups affect policy and 
evaluation at the state and, consequently, at the'local schoo,V 
district leveTs. . ^ ^ ' " . * ' 

Although the paper focuses primarily on the SCDE'^and tfte ^ 
^interactions between it and other^ agencies, "iny perceptions ofT:hose 
'interactions are no doubt colored' not only by- my particular , ^ 
responsibilities in the agency, ,but al^o as a result of seven years 
as an administrator in the Wyoming Oepartment of Education and nine 
years as a^acher in public schools, i a Mi ssissippi and Wyoming. An 
external oflirver, or other ^taff of the SCDE, might have 
perceptions of the pol icy/ evaluation Interaction" that are quite 
different from my own. Since such differing perceptions are the 
rule rather than the exception, the writer recognizes .that the 
thoughts expressed herein are but one version of the "truth". 



• Although the tenns "pol^cy^^and "evaluation" are widely used; 
and;perhaps just as v/idely ^understood, it seems advisable to define 
the tef?ns>es tKey are 'used in the remainder of the paper* 'Policy, 
as used^erein, is defined as'.including all le^'slation, 
regulatians, ppsi tion , statements and policy statements, e.g.., "the 
expressed policy of .the State Board' of Education is — ", the 
intendei^purpose of whichvis to determine a course of action, * - ^ 
".estabri-IR a program^, or provide aN.framework within which decisions 
are to be made. ^ ^ * 

Ev^aluation, as used herein, U defined as the utilization of 
informat'ion', obtained through a sy&tematic process of data 
collection, for any of the fallow,ing purposes: assessing the impact 
of establishedvpolicies or programs; comparing the effectiveness of 
two or more.pragrans;' assessing the degree of compliance with 
established palicy; or influenci ng the e'stabl.i-shment of ^new or 
revised policies or- prog rams •'-J'h is definitiof^ of evaTua'tion 
delib^erately avbids any att'empt to draw the traditional academic 
distinctions b^eftween' pol icy^ stjudies,. research, evaluation, and * ; 
assessment, [his is done for iwo reasons^ first ^, the common' ^ ' 
distinctions^mtJng^hesa terms focus more on questi/ns asked and 
procedures^! low^fJ than on the use(s) made of th^ information 
collected i.e., the distiijctions ^re more semantic, and academic 

' than they are real and second, the result^s of research, evaluation 
(in the traditional sfense), assessment and policy studies are all 
used, ^t^ varying degrees, in efforts to formulate or modify policies 
and^programs. Whether a particular data collection effort should 
legitimately be labeled an ev.aluation, seems 'to be more 
appropriately determi/ied ,by the use(s) made of the data than Dy the 
particular study design or the procedures used in collecting the 
data. Regardless of the complexity and/or degree of sophistication 
of ap "evaluation^* de^^ign,. the act of 'collecting data does not 

^ constitute evaluation. Evaluation occurs only after the data are 
collected and then, only if the data are used as 'a basis for making^ 
j.udgenents about worth, va^lue or effectiveness.^ Although they may - 
not be acted upon by those in poUcy ^setting positj^ns, the first 
place such judgements are nonnaLly identifiable is in the evaluation 
report. ^ , • / , * 

Although they are consistent with the definition. given here, * 
Rjany of the -eSJampl es of eval uatlon.jusedjfi this, paper wi IT not be 
regarded by acadenicians as '"true" evaluation. However, the broad 
definition of-evaluatlon. previQusly gj-ven is' necessary in order .to 
-undei^tand the* policy/ evaluation interact'ion. ^ 

^ * Organization of P^Pei^ ^ 

The remainder of th6 paper is comprised of four major sections. 
'The first, "South Carolina Public Educatiorv: State Organization and 
Administration", provides a description of the context within which^ 
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t he_wr i t&g^j^^erce^U^ the po U cy/ ov a 1 ua t i a fM-fvter^rclrn}7r^^ 
"^n foi*iulated. The second, "Policyr Influence on Evaluation", 
concerns the ways in which policy determines what is to be evaluate* 
and the impact* that policy has on'eval uation methodology. The 
thjrd, "Evaluation: Influence on Policy", represents the ^^iter's 
perception of the conditions under which evaluation does,- and does 
not, influence policy. The final section, "Other Factors 
Influencing Policy and Evaluation", examines J;he impact which 
special interest groups have on the fonnul ationNif policy, the 
design of evaluation, and the uses to which evaluat^pn findings are 
put . t V 



' SOUTH CAROLINA PUBLIC EDUCATLON: 
STATE ORGANIZATION AND ADMINISTRATION 

. - ' State Superintendent of Education 

The Office of Stat§r Superintendent of Education was estaolisned 
Dy the Constitution of 1868. The Superintendent is elected on a 
partisan ballot ipr a four year tenn and therj^ is no limit on the 
number of consecutive terms which the Superintendent inay serve. 
During the period from reconstruction until 1579, twelve 
superintendents were elected to office. The current Superintendent, 
Dr. Charlie 6. Williams, began his first term in January, 1979, 

The general duties of the State Superintendent, as prescribed in 
the School La|^ of South Carolina (19751, include: ' 

1) serving as secretary and administrative officer 
to the State Board of Education; 

2) supervising and^managing Sll puoHc school fufWs 
provided by the St^te and Federal Soverrvnfents; 

3) organizing, 'staffing, and administering a State 
Department of Education; and 

4) administering, through the State Department of 
Education, all policies .and procedures 'adopted by the 
State Board of Education. 



^ State Boird of Education 

TFTe State Board of Education is comprised of seventeen memberj^^ 
one from efach of the sixteen judicial circuits and one mejnber at 
large. The members from judicial circuits. are elected by the 
legislative delegations representing the^counties of each circuit. 
The "at large" member is appointed by the Governor. The tenms of^ 
the manbers are four years and no menber may serve consecutive terms 
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except by the unanimous consent of all members of the county , 
legi slati^e delegations from his/her judicial ciruit. The statute 
pertaining to the composition of the State Board contains no 
provisions excluding professional educators from service on the 
Board. The present chairman, and five other members are 
professional educators, Althoagh the members are elected by their 
legisVative delegations, the practice of elating educator.s has 
recerived criticism from some Inegibers of the General Assembly. 
During the past several years, legtslation has been introduced, but 
not enacted, to restrict State BoaY*d membership to the lay public. 

' The general powers of the Board include: 

1) adopting policies, rules and regulations not 
inconsistent with the laws of the" State for its own 
goverf¥nent and for the*government of the public schools; 

2) annually approving budget requests for the 
institutions, agencies and services under the control of 
the Board; 

^ , 3)^ adopting, mioinium standards^ for any phase of 

education," as are considiered necessary to'a-id in 
providing adequate educational* opportunities and 
faci^ ities; 

4) prescribing and enforcing, rules for the examination and 
certification of teachers; and 

5) prescribing and enforcing courses of study for 
the public schools. 

4 

I State Department of Education 

The administrative structure of ^ the State Department of 
Education includes three divisions which;^are under the supervision 
of Deputy Superintendents wha ar^ directly responsible to the State 
Superintendent of ^Education,^ Each of the three divisions, 
Administration and Planning, Instruction, and Finance and 
Oper'ation^, Includes Several offices' which, collectively, administer' 
the progr^ams for which the agency fs responsible. ^ Although'most of 
the of-fices include two or more sections, t)ie organizational chart 
(Figure 1) .does not iticlude detail below the office level. The 
organizational pattern of the Department has remained relatively 
stable during the five years in which the writer has bean an 
employee.. The only significant changes, the creation of the ^ 
positions for associate superintendents and special assistant for 
legislative affairs, as well as plaqing the Office of Personnel 
under the direct supervision of -the- State Superintendent, have 
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occurred since January, 1979.1 

The Department employs 1079 individuals of whom approximately 
one-half are involved in the maintenance and operation of the state 
supported pupil transportation system. With the exception of the 
employees of the Office of Transportation, most of the staff are 
based in the agency offices in Columbia. 

Evaluation functions within the agency are decentrafized. 
Although^no office title within the agency includes the word 
evaluation, several offices carry out activities which fall withi/i 
the .broad definition of evaluation which was presented earlier. 
There is, however, no State Board or agency-wide policy concerning 
evaluation responsibilities, *s 

Each office which funds programs operated by school districts 
has, or assumes, the responsibility for evaluating, or monitoring' 
the evaluation of, all programs which it administers. The 
determination of 'i^ich programs are actually evaluated is, more 
often than not, a function of federal mandates. The of f ices most 
heavily impacted by federal mandates for evaluation are Federal 
Programs, Adult Education, Vocational Education, and Programs for 
the Handicapped, x - 

Two offices which do not administer funds for locally operated 
programs but which are involved in evaluation activities are 
Technical Assistance and Surveys, and Research. The Office of . 
Technical Assistance and Surveys conducts, at the request of local 
school boards, studies to determine needs in the areas of 
administration, curriculum, personnel, and 'facil itias. The results 
of these studies frequently provide the basis for district planning 
to meet the td^ntifted needs. Although in the traditional sense 
these studies might not be considered evaluations, they do provide 
information on which pdlicy is based and actions are taken.. 

The office of Research is involved in evaluation activities in 
three ways. The ,first,'and most time consuming, is through the 
administration of the Statewide Testing and Basic Skil]s Assessment 



iThe State Superintendent recently initiated a management ' 
reviev/ of the agency for the purpose of determining whether sane 
organizational changes may be desii^able in order to more effectively 
and efficiently fulfill the responsibilities of leadership, service, 
and regulatioi>. ^^s a result of the study, 'the organizational chaft 
(Figure 1) may be inaccurate by the time this article appears in 
print. With that possibility in mind, I have attempted to keep my 
observations specific to the agency (as requested) while at th^ same 
time keeping the interpretations general. 
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Programs. The second involves data collection to assess the 
financial imp|ct of proposed or existing policies, e.g.', "What is 
tjje projected cost for facilities required to implement a 

egislaiTve mandate to reduce pupil/teAher ratios in grades one 
through three?". Finally^ evaluations are occassional ly conducted 
at the ic;equest of other offices within the agency or as a result of 
a decision of the State Superintendent. 

PJDLICY: INFLUENCE ON EVALUATION 

Organizatronal Influence 



Whether the decentralization of evaluation. responsibilities in 
the SCDE is more a function of default than ^onsc ious deci^sion 
making is not knQwn. However, evaluation as a recogni-zed 
responsibll ity of State Education Agencies is still in its 
childhood, or at best early adolescence, and some agencies have not 
chosen to organize in a manner that concentrates that responsibility 
in one unit. Experience gained by serving in two state education 
agencies, one in wKich the evaluation function is centralized and 
one m which it is decentralized, indicates that both organizational 
patterns have their unique disadvantages. 

Jhe influence of Agency structure on evaluation is evidenced in 
several ways. When the responsibilities for evaluation are 
decentralized there may be no canmon criteria which are uniformly 
appl led ^either in the employment of staff or in the design of 
evaluations. Thi s -frequently results in: - • 

1) CQnsiderable variation in the level of expertise, 

of evaluatiort staff assigned td) various offices within 
the agency; 

» 

2) a greater than acceptabl e degree of variation 
* in the quality of evaluations; and 

3) a lack of consistency in the kinds of evaluation 
requirement s whi ch the various offices impose on local 
school 'di stricts. ' 

^hen eval uatiOTi-responsibilities are decentralized, evaluators 
are frequently directly responsible to. the administrators of the ^ 
programs for which they have evaluative responsibility. 'Even if 
objectivity can be maintained in such situations, evaluation 
findings may 1 ack -credibility because of an apparent Conflict of 
interest.' This situation, however,, is not automatical ly overcome by 
centralizing the .ev.al uation function within the agency. Incjividuals 
or groups external to the agency may still consider findings with 
which they disagree to be merely a.reflectlon of the agency* s bias. 
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With centralization of the. evaluation function, the 
disadvantages cited above are eliminated or at least allevi^ted^ On 
the other hand, if evaluators are assigned to a unit that has no 
progransnatic responsibility, they may well be viewed with suspicion 
and distrust by t he administr ators of those programs which are to be 
evaluated. Additionally, clxnmunications across or through 
administrative channels in. a bureaucracy can be both slow and 
frustrating. 

Whether responsibilities for evaluation are centralized or 
decentralized is probably less important than having an agency 
commitment to "good" evaluation for the purpose of addressing policy 
relevant issues. In the absence of such a cornmit'nent, it is 
unlikely that agencies will anticipate informatio/i needs and ' 
consequently will frequently be placed in a reactive rather than 
proactive role in the policy making process. 



Policy, regardless of the governmental level at which it is 
creai^^apparently" influences evaluation in two major ways: first 
by deferfiiining wtfat is to be evaluated, and second, by shaping or 
dictating the evaluation methodology. 

Oetermining^^it is to be evaluated is, in many instances, not a 
matter of choice for any State Department of Education since State 
and Federal statutes and regulations, e.g., ESEA Title I, may be 
very elicit .in this regard. A^major portion of the evaluation 
efforts of the SCOE are directed^tcrward complying'with federal 
mandates although federal funds account for only approximatl ey 14% 
of the annual expenditures for public elementary and secondary 
education in South Carolina. This is not to say that too many 
resources are expended in evaluating the effectiveness of federally 
funded prograi^s. It does suggest, however, that in the past have 
probably spent^r too little time and money in evaluating state 
funded prograns* 



The impact that policy has on shaping of^determining evaluation 
methodology is nowhere^more easily identifiable than in the 
evaluation requirements for ESEA Title I. 'The federal regulations 
stipulate the only evaluation methodologies that may be used by 
states and local school districts. 'Any exceptions to the prescribed 
methodologies must be approved by the U.S/ Commi ssioner of 
Education^ The expressed rationale for these models is that they 
will yield comparable data on pupil achievement that can be 
aggregated to the national l^evel, i.e., across school districts and 
states. Various critics have raised questions about the. validity Qf 
this assertion because of some Onresolved technical Issues 
surrounding the models. Assuming, however, that the models can and 



Federal Influence 



149 



ICO 



do yield reliable and valid data, another basic question is still 
unanswered. How can aggregated pupil achieverifent data be used in 
addressing a policy issue which, on the face of it, is more^socfal 
and political than educational in nature? Would it not be 
sufficient, and perhaps more appropriate, to determine: 1) whether 
the target population as defined in legislation is actually being 
served; and 2) what type of instructional programs or what 
organizational patterns are most effective in meeting the needs of 
the educationally disadvantaged? ^ 

As a member of the Evaluation Sub-Committee of the Committee on 
Evaluation and Information Systems (CEIS) of the Council of Chief 
State^School Officers, the writer has ^een priviledged to hear much 
of the debate concerning the Title Tmodels, ApajTt from the 
questions 'which have been raised, about the t^dirj^cal quality of the 
models and the policy relevance of the data, concerns have also been 
voiced about the "test only" approach to evaluation. Local district 
personnel are concerned that the use of Title I funds for evaluation 
will b€ restricted to the collection of data required by feaeral 
regulations and consequently, their efforts to examine the 
effectiveness of other progr^ ccmponents will be severely hampered. 

In addition to the Title I evaluation modefs ^ich are already 
being use^, the U.S.O,.E. is developing models for the "eval uation of 
programs for mi^grants and for children in institutions for the 
neglected and delinoijent (N and D).^ Jtp the first draft stage, the\ 
models for programs^for N and D are* ^o .pupil achieveipent based. 
The methodology tolbe used is certainly being determijied by policy 
which will probabl^sJ)^ promul gat^d in -the form of regulations. 

Beyond the impact that specific program policy, e,g,. Title I, 
may have on evaluation methodology as applied to that program, the 
effects oft^n carry over into other programs. For exampje, the 
comparability and non-supplanting requirements of Title I, coupled 
with the Office of Civil Ri-ghts regulations v/hich prohibit grouping 
that results in the fonmation of racially identifiable classes, 
virtually prohibits the use of experimental or quasi-experimentil 
design 1n evaluating programs that' may have little, if any, 
relationship to the federal programs which have placed constraints 
on evaluation In general. 



State Influellce 

Three recently enacted South Carolina statutes pertaining to 
education include explicit evaluation requirements. The?e are the 
"South Carolina Education Finance Act of 1977"; the "Basic Skills 
Assessment Act of 1978"; and the "Teacher Training, Evaluation and 
Certification Act of 1979". 
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The Education Finance Act includes an-"accoufitabil ity" section 
which requires: 

1) the establishment of school advisory councils; 

2) school and district based needs assessments; 

3) tjKe dev.elopment of annual plans to \neet Identified 
needs; 

4) district participation in the statewide 
testing program as prescribed by the' State Board 
of Education; and 

■ 

5) annual reporting of program effectiveness to the general 
public ^nd the State Board. 

*, 

The Basic Skills Assessment Act, although not as expliirrt^in its 
requirements as .are the federal regulations concerning Title I, 
contains reqgireinents whic^reffect ively shape methodology and 
instrujnentation* The law requires: 

1) the administration of a readiness test at the 
beginning of grade one;^ 

2) tests of reading and mathematics at the end of 
** gr'ades one, two, and three; 

3) tests of reading, mathematics, and writing at 
the end of grades 6 and 8; and - , 

4) a test of "Adult Functional Competency" at the 
end of Grade 11. 

The law includes ci time-table for the development and ifnpl ementation 
of the program and stipljlates that the tests shall be ' 
criterion-referenced. -further, the tests shall be used for the 
purpose of diagnosing student deficiencies and as a basis for 
remediati6n. The tests ar^ not to be used as a basis for proinotion 
or non- promotion. None of these stipulations are necessarily 
undesirable, but'they do have considerable impact in shaping the 
assessment program. 

The Teacher Training, Evaluation, and Certification Act mandates 
major changes in teacher 'training and certification procedures* The 
requirements of the legislation include: 

1) all applicants for admission to teacher education 
• programs in State supported -^insti tutions must 
Successfully complete a basic skitls examination in 
reading, writing, and mathematics; 
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2) ^"the development of an" instrument to be" used by . 
• colleges and universities in evaluating all student 

teachers; 

V* 

3) development of an Instrument to^ be used by local 
school' districts^) evaluating teachers during their 

# provisional year of certification; 

4) successful' canpl.etion of a teaching area / 
examination as-^ne requirement for provis^r^nal 

/ certification; and ^ 

5) discontinuing the use, after July 1, 1981, of 

^ _ the Commons: Examinatio/i of the National Teacher 

Examinations *for the purpose of teacher certifica- 
tion. 

The legislation includes a number of other provisions but those 
cited appear to be the one3 which impact most significantly on 
evaluation procedures and methodology.* 

Not only does policy influence evaluation by determining what is 
to be evaluated and what methodology may be used, it also has Impact 
on local acc'eptance of programs. Local district personnel 

^frequently perceive evaluation requirements as being somewhat 
arbitrary and infringing upon their rights to make decisions locally 
regarding curriculum and instruction. Questions. that are of 

'interest to funding agencies (staj^e-- and, federal) may be pf .little , 
if any, interest to local district administrators. For /exanrpl e, few 
district administrators are interested in or ctyficerned wth the 
external validity of a project or program^ which is operating within 

\their school district.. Whether the projector program works in 
the>c particular district is of more concern than whether it may be 
exportable to some othec district similarly situated. Evaluation 
design -^i^^ however, determined by the questions of interest to the 
funding agency and ext-ernal validity is of interest if the agency is 
considering replication of the project. Although the funding 
agencie$ may ask legitimate questions, lack of sensitivity to local 
need^ does 1 ittl e to gain the support and cooperation of Ixal 
district personnel who are implementing the program* 

All evaluations are not initiated as a direct result of 
statutes, rules, regulations, or other written ^pol icy statements 
which may require that evaluations be conducted* Occassional ly , 
evaluations are requested long after a course of action has been 
determined and a program' has been implemented. Such evaluations are 
usually sought as a mearts of either providing data 'to generate 
continued support for a decision that was initially based primarily 
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on beliefs or for generating support* for the expansion of the 
pro(jram. Jhis always raisers the question of whether what is being 
sought is really evaluation or a "Good-Housekeeping seal of * 
approval". There is no intended implication that those^vho seek 
such "legitimatizing" evaluations are dishonest or unethical, Td 
th§ contrary, they are almost always Sincere, dedicated individuals 
who firmly believe- that their program is working, has great merit, 
and should either be maintained or expanded. Seldom have these 
"sta):e-holders" entertained the possibility prior to conducting ^js^ 
evaluation, that the results may ncgi Support thgir^biases. 1 
• ft " 

Although the questions addressed by such "legitimatizing" 
evaluations are not directly influenced by the pol icies oo^^actiori^ 
that established the programs being evaluated, the methd^logy M 
certa'inly influenced by their ex post facto nature. ^ 



E'VALUMIGN: INFLUENCE ON^POLICY 

. A cofmion lament of evaluators is that* the resul ts of their 
efforts are not used by decision makers. Although this is not 
always the case, the situation occurs frequently enough to cause 
great concern among those who practi'ce the art (or science) of 
•evaluation. Assuming that* deci sion makers are reasonaoly ratienal 
Hndi-viduals who would prefer to make decisions on the basis of 
^information rat^her than intuition, there must be reasons why 
evaluation findings are not used. Experience' indie at^es several'* 
possibil i't>es^ 

1) the conc>usions of the eva^uator are not germane 
to the deci sio/^Sx that must"'be m^'dei 

" ) 

2) the evaluation findings are, not reported irwa 
manner that communicates to the policy maKers,-* 

3) .the^f indings do not support the biases of those 

makTng po^tcy decisions; or 

4) a lack of credibil ity Vesul ts either from poorTy 
conceived or conduct^ed evaluation or the reporting 
agency no^eing credible to the potential' users 
of the d^j^. 

£ach. of these possible reasons f9r ignoring the results of 
evaluation is expanded to some degre^ in the remainder of this 
section. * * ife . 



^ ' ' , ' Conclusions Not Germane 

/' * * • 

' Educational evaluators tend to function \x\ the manner* that thfi 
label impiie?, i.e., given freedom in designing evaluations they 
tend to focus only on the educational and-cost a,spects of program qv. 
. policy and usually ignore', or'fail to realize, that'there are oftep^ 
• social and political, as well as educational and financial issuesr * 
involved. The ^yaluatior> of any program or policy may well be o^* 
interest to more^han one group of decision makers and these various 
groups may legitimately be interested in answers to different" 
' que$tions•^For example, the local district administrator of a 
federally funded program may be interested .only in whether the 
program produces gains in student achievement beyond that whicfi 
would be predicted without pr.ogram intei»vention. The state level 
administrator's interest may extend to the ijuestion of external 
validity. Administrators at the federal level may share^the \ 
interests of state and local administrators, but in addition they 
want data that ca/i be aggregated across states, whereas,, the /ugding 
body. Congress may be more* interested in the social ^and 'pdljticll 
.aspects of the progr^, e.g., is, the group for which the program, v/as 
intended actually be^ng served? Compensatory education programs are. 
a^case in point, /her^e is serious doubt tha^ Congress' wil 1 ignore 
social and political issues and make significant changes in any 
ccxnpens^ory education, programs solely on the basis of^pu'pil 
achievement data as a measure of prograin effectivenefl. 

Evaluations which are too naVrowly focused yield results which 
may.be viewed « not germane to the issue to be decided* * An example, 
is available as a result of an action by" the South Carolina General 
Asseinbly which' provided funding for a pilot program to reduce 
pupil/teacher ratio in the /first grade from 26:1 to 20:1. The 
NOffice of Research in the SCDE was assigned the responsibility for 
evaluating' the' pi'lot program. Since the intent of the, pilot program', 
was "obviously" to determine the cost-effectiveness of a reduction 
in pupil/teacher ratio^as a means of increasing student achievement, 
an evaluattcn focused on achi evement /gain was designed and conducted 
over a two-year period*. The results 'of the evaluation Werfe 
consistent with a large body of the literati^re on the subject and 
indicated that a reduction in pupil/teacher ratio of the magnitude 
involved wa^#iot a co^t effective means of increasing student 
achievefjient^. Shortly after the evaluation results^wfere released. 



2/\mong the common problems enceuntered' by^ evaluatcfrs arO:hose 
created by insufficient time and financing. ^ If it had been [rossible 
to^conduct a longitudinal study of 'the students involved in the 
prbgr^, the results mignt have been different. Although the 
educational system is generally concerned with long-term benefits, 
evaluation-is frequently restricted to tlje^examination of immediate 
outcomes. 
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the General Assembly enacted the Education F,inance Act of 1911 and 
'mandated, a reduction of pupiT/teacher rat4o from 26:1 to 20:1 in 
grades one through three. ,Why? In retrospect, tTie evaluation was 
too narrowly focused and failed to t^ke into consideration the 
political aspects of the' issue. The South Carolina Education 
Association, a^rather effective lobbying' group^, was exerting * 
pressure on the legislature and , as a means of gainihg their support 
for the total, finance bill, the proviso relating to pupil/teacher 
ratio wa|" includecl. The questipn addressed in. the evaluation - 
certainly seems to be a 1 egitimate one but perhaps the utility of 
the study coyld have been improved by including a survey of teachers 
to determine whether they preferred to have the available funds used 
to reduce puqiT/teacher ratio or to provide an Increase irT'salary. 
Additionally, taxpayers, especially those whd are parents of 
school-age children, may be supportive of smaller, classes because of 
* their beliefs that children receive more individual attention in 
small^er classes. The point is, there were other questions, in , 
aadition^rto thgse related to gain scores, that probably would have 
been of interest to the pol icy-makers and, consequently, §hould have 
been incladed in the e^^aluation. 

If 'we are really interested in increasing the use of evaluation 
findings -in determining policy, more attention must be given to 
identi^fying the social and political, along with the educational -and 
economic, issues involved. No one is so .naive as to believe that 
^ ;all decisions affecting education are' made by educators , however, 
« the narrow focus of many evaluation designs is not consistent with 
^ur knowl edge' of the decision making process. The answer to the 
question "By whom will the-results be used?" is too often left until 
the finalreport is being written, in which case, a likely answer is 
"no one". Lf _eval uations are to yield policy relevant results, the 
various stake-holders and the questions of interest to them must be 
identifj'ed before the fac^. / ' 

' Evaluation Reports That Do Not Communicate 

\ ^ Thq policy maker who has the technical backg^ound requisite to 
the interpretation of the typical evaluation report is about as rare 
a specimen as a 5*2" professional basketball player. All too 
frequentlj^^ evaluaftors appear to be more interested in producing % 
repor;J:s ,for journals or discussions with their 'col leagues than in 
canmunicati^ng results to potential^ users of the data. The 
development of highly technical re"ports may work wonders for the 
ev*aluator*s ego-but the intended users ^ay be left with two choices: 
1) ignore the report, or 2) request assistance in interpreting it. 
A colleague once made a learned, but incomprehensible, presentation 





to a lay advisory canniittee. Following his oac-hour inonologue, 
during which the committee menbers were attentive and polite, two 
canmittee memoers made comments. The first, a newspaper editor, 
said, "I consider myself to be a reasonably intelligent man, but for 
the past hour you have insulted my intelligence by subjecting me'to 
your jargon." The second, a wealthy rancher, commented, "As a 
former schoo] board m§mbar, I made it a practice never to fund 
anything that I could no{; understand." Unfortunately, the message 
the committee members were trying to convey was lost on the speaker. 
The point is, that as long as we evaluators focus our communication 
efforts primarily on each other and the academic. community, we have 
little ro'^an for compla/nt when the results of our effort's are not 
used by decision makers. 

The Office of Research in the SCDE invested considerable time 
and^ffort in determining correlates of ^'achievement test scores m 
order to generate predicted school mea^n achievement scores for ^ 
compariscVi with those obtained through the Statewide Testing 
Program. The intended use was to stimulate local school ^ 
adminTstrators to ^ake a closer look at schools in which there were 
significant differences between the predicted and oDtained scores. 
The Office prepared and disseminated to district super fntendents a 
"non-technical" sijmmary report on the project. , One rfecipient of the 
report wrote to the State Superintendent Of Education and commented 
that if he was real ly 'expected to understand the report, it was 
probably a good thing that he had only^ a snort time left ppior to 
retirement. For those of us involved irr^preparing the report, the 
easiest thing to do was to agree that it was probaoTy good, that his 
time of retirement was near. In reality, however, he was^ probably 
not t^e only one of the ninety-two superintendents with whom we 
failed to communicate. The reports have since been revised and are 
nowi>eirfg used as initially intended in many of the South Carolina 
school districts. Unfortunately, we do not know the extent to whicli 
tfte potential utili^ty of the information was reduced by our initial 
failure to communicate. 



Results That Oo Not Support Stake-Holder Bias 

Precedent to a decisic^^o evaluate the impact of any program or 
policy is the decision that^reated the prpgr^ or formul ated^the 
policy. This decision usual reflects some belief of the policw^ 
makers tHaL^the course of aqtion being pursued is appropriate ano^ 
therefore ^uld yield beneflicial results. In essence, the 
stake-holders have evaluated", wi th .positive results, the course olF 
action before pursuing ity Consequently when formal evaluations 
result in findings cont^ry'to the policy makers' bias, the * 
/sra^ator-, and hi s/hpr^^ndings , may encounter considerable 
jresi stance* ^ The" ego inVoXvement -of a policy maker may be so great 
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that either revising an existing policy or formulating a new policy 
contrary to the prevlousy selected position represents a cost which 
is too great to t>e paid. ' In this case the policy maker may simply 
di sregard -the evaluation results and proceed oh the previously 
established course. 'On the other hand, the initial rejection of the 
findings may, with time, give way to a change In position. The 
writer has observed this shift occur so gradually that It was 
virtually impossible to determine the point at which th^ change was 
made. (This seems to ha^ve implications for studies of evaluation 
utilization. In what time-frame must the results be used In order 
to^consider them "utilized"?). 

For several years the SCDE supported the. imp] anentati on of 
extended-day kindergarten programs^ln the belief that sucn programs 
were efficacious in developing "readiness" for first grade. When 
the results of a third-party evaluation Indicated non-signi f icant 
differences between half-day and extended-day, programs m this 
regard, the findings met with considerable resistance and 
unjustifiable questions were raised about the technical quality of 
-ctfe study. Gradually, however, the Drogram administrators have 
shifted tneir position on the issue. The point to oe made is that 
it Riay be unreasonable to expect negative evaluation findings to oe 
immrnediately and^wamly embracjjiLbj those^who are stake-holders in 
the program. Perhaps the best that can be hoped for Is that when 
decision makei/s are given both information and ulme, reason will 

prevail. ^^-^ c ^ ^ ^ ^ 

*/ 

Lack of Credibi 1 ity 
7^ ^ 

Ai^--stated earlier, non-utilization of evalua^tion f IndiHU^^due to' 
lack of credibility may possib^ly be attributed to one of two^ 
reasons; l)'the evaluation was poorly conceived and/or conducted, 
or 2) the agency reporting the^ results lacks credibility with the ' 
potential user's of the data. The only apparent defense against the 
first of these is to conduct other evaluations and not make the same 
mistake twice. Unless, however", the services of a different 
evaluator are- util ized, the results may still De viewed with 
suspicion by the target audience. 

In the second' instance (lack of credibility) there, again, Is 
little that, can be done after the fact. Pri-or to engaging in any 
evalua^tion effort, the agency should carefully consider whether the 
potential users. of the data ar,e likely to consider the results to be 
objective and unbiased. 'When the evaluation function is centralized 
within the Department of Education and, consecjlfer^tly , evaluators are 
not answ^able to program administrators, evaluation results may- 
have credibility within the agerfcy itself. > On the other hand,', when 
v'iew^ externally, even a centralized evaluation function is still a 
part of the total agency and results may be suspect. Unless, the 
results of the evaluation are solely for internal decision making, 
thi s' probably argues against having any State Department of 
Education evaluate programs which it administers. 
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Other Factors Influencing Policy and Evaluation 

Although the influence which special' interest -groups have upon 
the formulatiXDn of policy and, consequently, upon evaluation has 
been mentioned earlier, the magnitude of this influence is^such that 
it seems to warrant some special attention. Policy is not 
formufated in a vacuum. Policy' is formulated only in response to'^ 
external influence or, more to the pointy in response to some real 
or imagined need. In the opinion of the writer, any discuss^ion of 
pol icy/eval uation interaction which does not consider those external 
influences ignores the force which drives the system. Regardless of 
whether a policy is developed by the Congress, the State 
Legislature, the 'State Board of Education, or the State 
Superintendent^ it appears that the policy is always in response to 
the stated, or inferred needs or desires, of some group. For 
example, members of the legislature, as duly glecied representatives, 
of th^ public, enact statutes (educational and otherwise) which, at^ 
least in theory, reflect the desires and interests of the electors. 
The State Board of Education then translates the education related 
statutes into rules and regulations which are administered by the 
State Superintendent of Education* In turn, local boards of 
education may develop their own policies designed to implement those 
imposed jjpon them. 

Any of the pol icy-makf^m groups, the legislature, the State 
Beard *bf^ Education , or locaKiilbool boards may generate policy in 
response to their own constituents wi thout, being motivated to do so 
by one olr t'be higher level policy-making bod-ies. In, any case, 
howe^ver, the link between external influences and policy seems 
obvious. . 

^flhis translates, then to a system of interaction that might be 
ortrayed as in Figure 2. The implications are that special - 
terests directly infl uence ^pol icy and^ indirectly, evaluation; 
olicy directly influences evaluation; and eva1uat4on may influence 
policy either directly by feedback to the policy-makers or 
indirectly through feedback to spec-ial interests. 
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Figure ^2 
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Unfortunately, the process is not nearly so straightforward as 
the preceding paragraphs and Figure 2 my imply. There seem to be' 
at least two sources of "noise" in the system that have implications 
for evaluation §nd the use of evaluation findings. First, there are 
usually competing interests 3t work any time that policy (law, rule, 
or regijlation) is established-. Consequently, canpromise may be made 
in framing the policy with the result that no one is completely 
satisfied with the outcome. If this is true, various 
"stake-holders" may seek quite different results from evaluations or 
at lea^st may have different biases toward those results. 'Second, 
even if the policy.il a^ precise statement of the expressed desires 
of the interest group, the underlying intent ;n3y not be reflecte<i in 
the po.licy. This may lead evaluators down the primrose path to 
asking the^//rong questions (or at least not enough questions) if 
they mistakenly assume that a policy i^ addressing an educational 
issue when, in fact, the force behind the policy was more of a 
social , political, or economic nature. . The earlier reference to 
the question of reducing pupil/teacher ratio provides an example of 
the pitfalls which can b^ encountered hs a result of either 
ignoring, or failing to recognize the motivation for a particular 
pol icy. , ' ^ 

In concluding, it seems desirable to provide an example of the 
special interest/policy/ evaluation interaction. In using the South 
Carolina Basic Skills Assessment Act; of 1978 as an example, the 
writer is aware that his perceptions constitute only one version of 
the "truth", 

Jhe legislative history of the act began in 1977 when two 
representatives, both members of the Black Caucus, co-sponsored a 
bill cal-ling for the implementation of^a program of grade by /grade 
promotion based on achievement test scores. The bill provided that 
the State Department of Education ,would select the tests and 
determine the required scores. The ^propos^d program was to be 
implemented 'over a- twelve year period beginning with grade one and 
adding a grade each year until all grades were included. The bill 
did not r^ecei^ favorable action j.n the General Assembly but it did 
generate sufficient interest to resul t^^TtrTa joint resolution 
creating a special committee to st.udy the issue of minimum 
competertcy in the basic skills, , 

In its report ''to the General Assembly, the comrnlttfee included a 
drafts of proposed legislation. Al though- over eighty amendments were 
introduced during the passage of the bill in 1978, the law is 
essiehtially the same^as the bill proposed by the committee. There 
is, hdwever, one major difference* The correnlttee's prop^^sal that an 
eleventh grade test of "Adult Functi&nal Competency" be required for 
a high school diploma was deleted prior to passage of the act. What 
began in 1977 as a move to require grade by grade promotion based on 




testing, culminated in 1973 with a required testing program to be 
used for the diagnosis of student deficiencies and as a basis for 
providing basic instruction to assist students in ovgrcoming those 
deficiencies. Major comproaii ses , the result of the interests of 
various groups, were reached along the way. 

What motivated the introduction of the original Dill. in 1977? 
The most obvious answer is that it was simply an extension of the ^ 
national trend towards competency testing, Thet prime sponsor of the 
bill says that such was not the case. According to the sfjonsor 
there were two factors which prompted hiin to introduce the 
legislation. First, ''his constituents were concerned that many of 
their'chi Idren were only *5emi-l iterate upon graduation from high 
school and they were seeking a remedy for that situation. Second, 
•the legislator had access to data collected tnrough the S'CDE 
operated Statewide Testing Program which Indicated ttiat,- on the 
average, the achieve^ient 'scores of minority students were 
considerably below those of the majority. He attributed these 
differences to poor teaching and social promotion. He ^apparently 
perceived profjiotion as a function of testing to be a renedy for the 
situation. 

The Basic Skills Assessment legi'slation is a rather significant 
policy statement concerning evaluation. Apparently u came Into 
bging as a result of the influence of special interests and tr\e use 
of available eval uation tiata that supported the concern of the 
interest group. * 



Conclusion 




This paper was written for the purpose of^providing a 
non-aoademic perspective of the pol icy/evaltfatidn interaction in a' 
state education agency. While I am conf ident\(at the 99% level) 
that it IS non-academic, I am not equally confident that it provides 
a '*true" picture of the interaction which was the suDject of the 
paper. As stated earlier, others may/ and profrHbJv do, have quite 
different perceptions of the ways in which policyiiB^' evaluation 
interact^ or^fail to do so, within the SCDE. But that is the, nature 
of evaluation, take the information and make your own judgement. 
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POLICY AND EVALUATION; 
A CONCEPTUAL STUDY* 

Thomas F. Green 
Syracuse University 



What is {or can be) the relation between public^ policy , on 
the one h.andy and evaluation, on the other? Is there a way to 
attain clarity in understanding what an appropriate relationship 
would be? I intend to answer these (Questions in twQ steps. I 
shall consider first the typical character of policy questions. 
In a second part, I shall examine different facets of the policy* 
process — analysis-, formation, decision, and political judgment. ' 
I believe that these steps, taken toge-^Hl will allow us to^form 
a^view, however tentative, about the iMevance of evaluation as a 
professional practice to various aspects of public policy. 



^*The numbered footnotes in this paper are not *really »' 
footnotes. Thesn refer to corresponding items in the attached 
Appendix. This paper can be read as an independent analysis. It 
constitutes an attempt to reach conceptual clarity on certain 
features o*f th§ relation and interrelation between the practice 
of evaluation, on the one hand, and the formation of public 
policy, on the other. ' '* 

But the numbered items in the Appendix establish specific ^ 
Unj^ between this paper and a body of six others commissioned by 
the -fer thwest Regional Educational Laboratory dealing with- the 
practice of evaluation in State Education Departments. 

The "numbered items, then, are citations of points in those 
other papers that either provide the practical illustration of 
points made in ttre text or have actually provoked points made in 
the text. ^The underlying principie'is that a philosophical 
analysis of Suc5 matters should, on the one hand, illuminate what 
practitioners do and say about their work; but, on the other 
hand, *^i.{_^hould' also arise frojn a serious study of what they say 
and do. Philosophy, in this sense, is simply the explication of 
everyday life. _ 
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Policy Question^ ^ 

N 9 

There is probably no single definition of "policy" adequate 
to capture the fyll range of ordinary usage. Such a definition 
would have to satisfactorily capture the likenesses and 
differepces between manager ial. decisions , guides to practice, 
i^ules ^^legislation, basic choices of political direction, and 
the bar application of standard requirements in" administjration 
{No one gets unemployment assistance f^ more tl^ twenty-one 
- -</eeks) . It would have to include some matters that fall under 
••standard Operating Procedures'* (file expense accounts, within ten 
^ays with receipts), matters of personal practice (I don't answer 
the phone at home because it's never for me* Avoid arguments m 
the office. Don'< give G-3s unsatisfactory ratings; it takes too 
long to defend them.) Although each of these things can be 
called ''policy'', the tetm has^ in each case, a slightly different 
meaning. ^ . , ^ 

On the other hand, it is possible to establish ^me common 
features' of what we ordinarily take to constitute a policy 
question / especially if our concern -is with £ublic * pol icy , So I 
wish to rendec the question, "What is policy?" by asking "What is" 
a policy question?" ' , ' ' 

*A policy , question is a request for a line- of action aimed at 
—securing an optimal resolution of a conflict between different 

goods, all of 'which must be accepted, but which/ taken together, , 
. cannot all be raaximize^J* That is to say, we do not'have a 
' -wellr-formed policy question, a fully formulated statement of a 
policy problem^ until we are able to state the set of values or 
goods from which the question arises, and unless we are able to 
= state that set of .values or goods, so^.that we can discern their 
mutual inconsistency. 
^ ' "*'*^Th^ 'issues* involved in the contemporary^ movement for 'fiscal 
reform in education provide about as clear a model of policy 
^questions generally as it is possible to shape. The policy 
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issues are always "n'ested" within^ a set of mutually incompatible 
values or goods. We seek 

— (1) Equal educational opportunity tor children. 
(2) An equitable distr ibutioi^ of the tax burden. 
{3) Local control of education* 

(4) Responsible management of the State budget. ^ 

Maximizing any one of these goods — that is, getting as much of it 
as we can"will do damage to the advancement, of the others. The 
policy problem is generated by the fact that we accept all four 
of these aims and yet they cannot all be maximized. We cannot 
have all tne local contlrol possible because doing so will 
probably mean getting less than woul<3 be good in the way of 
equity for children ana taxpayers and control on the public 
budget. On the pther hand, if we maximize equity for children, 
then we are likely to get more inequity m the tax burden and 
less local control. The problems of educational finance policy, 
in short, do not arise merely from the need to establish a more 
equitable system for taxpayers and children* They arise ratttfer 
from the need to do so within a system^ of pubTic goods that 
secures also both local control and responsible public management 
I daresay that all issues that we would describe as questions 
of p^ublic (or even personal) policy have this feature,^ They 
are always "nested** in a set of social values or social goods 
which must ail be considered, but which, taken together, are more 
or less mutually incompatible. Consider the issues surrounding 
the imposition of exit standards at the secondary school. Here 
we seek the mutual benefits of: 

kX) Universal 'attainment (or at least racially and 
ethnically balanced frequencies of attainment)* 

(2) Common standards of achievement* 

(3) Culturally p,luralistic communities. 

The mutual inconsistency of these aims is transparent when 
thfey are visibly 3uxtaposed* The difficulties of f,inding some 
balance in thf entire set is well illustrated. by rfecent 
experience in {Florida and Virginia.^ P61icj.es are established 
to maximize common standards of achievement* The immediate 



consequence is to pay a price in securing the othe^\two goods. 
So the policy is adjusted — or its implementation is delayed. It 

^ is not unreasonable to surmise that continuing adjustments will 
result in policies that appear on the surface to depart 
substantially from previous formulations, but whose consequences 
arie not substantially different. Such a result is likely to 
eri^nate more from the presence of mutually incompatible values or 
good^ than from administrative "bungling", or blindness, or from 
ifferf iciency , or from political chicanery. That is to say, 
nerttrer^t)ie most efficient action nor the most technically 
profficient analysis will suffice to resolve the central conflict 
between the social aims within which the policy question 
resides. In general, there is no technical solution to a policy 

'question* For example, there is no purely technical resolution 
of the fact that if, in social institutiotis, we get all the 
efficiency we can, then we are likely to have less community than 
we need or desire. 

Triis conclusion, however, may seem outrageously facile. It 
deserves some explanation, and that explanation can be discovered 
in two pbints* The first requires that we grasp the important ^ 
fact that what counts as an answer to a policy question always 
takes the form of a "What we should do" and riever^a "What w^ 
know". Only practical questions are admissible in a public 
forum, never theoretical questions. And this is fortunate 
indeed* If means that in the domain of policy we are able to 
arrive ^at agreement on what to do without homing to agree on the 
reasons for doing^it. We must be able to agree on a'line. of 
action and s t^^k to it, even when we do not agree on what is good 
and even when we have different goals. ^ 

The result of a policy question is always a decision and an 
action. The result of a theoretical question is always a truth - 

• claim* Policy deliberation is aimed at action, not at^ the 
acquisition of knowledge; theoretical questions are always aimed 
at the acquisiti6n of knowledge, not at action* I do not mean b^. 
this claim that we can or ever should make public decisions 
without knowledge. Social action^ no doubt, should be informed. 
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Nor do I mean that we can ever gain greater knowledge without 

\ 

some action, ' Research, after all, is a kind of action, I mean 
only that wise policy is never made with enough knowledge to 
determine a decision, and' policy questions are never asked out 'of 
a primary interest in adding to our )cnowledge,^ 

In this day and age^ it is not* hard to imagine someone saying, 
**If we just had a -methodology sufficiently sophisticated and a 
body of relevant data sufficiently refined, then we could answer 
whatever , policy questions may com^ along." Such a person has 
been captured by a delusion. The delusion consists i/i supposing 
that a policy ques^tion is a theoretical question when, in fact, 
it is not. Any time we*suppose that a policy question can be 
resolved by some addition to our knowledge, then it will turn out 
tb^-t-iwhat we supposed was a question of policy has turned out to 
be merely a problem of engineering or efficient administration 
instead. * ^ 

My point then is not that wq shoula abandon all attempts to 
iinprove our methods of e^luation or policy analysis, *My point 
is ratl^er that since our indecisibn in matters of policy does not 
arise from th*e l^ck of such methods, therefore, it is unlakely to 
be laid to re3# by their development. In matters of policy,* we 
are confronted with indecision not because *our knowledge or 
technical facil*ity is faulty but precisely because we are 
confronted with a kind of question that, in principle , canrjot be 
answered by any increment or improvement of knowledge, Our'"^^^ 
answers to policy questions mfey be improved by" better information 
and better analyses in the sense that they will be more 
rationally persuasive. But such questions can, will, and usually 
are, answered even without such information. Furthermore, it is 
not obvious Jihat the answers given in the absence ot such 
analyses are worse than' or even often different from the answers 
that would be given in their presence. In short, the answers may 
be "^better grounded** rationally, but still not different o^ 
better io arty other sense. ^ Wfe can, no doubt, do something 
more rationally persuasive than consulting chicken entrails, but 
we^are unlikely to get anything that in its result's^'is quite as 
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decisive Tu^And^th is is so because of the properties. Of «policy 
questions, not because of deficiencies in policy evaluation. 

This observation brings me thert to a second important reason 
why there a!:e no technical solutions to policy "problems. In 
paradise, there is n6 policy — except,, per-haps^ admissions 
^Iii:ies, But why do policy questions not arise in paradise? 
There are-^ny ideas of .paradise / of course, but on the whole, | 
one IS inclinea to view that men conceive of it as a perfected 
slate of affairs in which wants presently denied in a*n impe-rfect 
world wi^l somewhere or' sometime be satisfied in a perfected 
^world, 'Consider the view Emerson" expressed in his essay "On 
Compensation"* K« heard a sermon, the chief message* of which wols 
that those saintly and good souls of the world who puffer without 
We corhiorts and amenities of life should p^^evere nonetheless 
ilili* ?^eir goodness.^ For -though sinners ,i!jwith their fine carriag'es 
"ar\d furs, may seem to prosper now, they shall suffer later; and 
tho\jgh saints may ha^£^ to do without, they shall later 'be" 

tded*. ^'Th^^mport of such a sermon, thought Emerson, was the, 
ge^Jo^ saint to sinnfer, "?ou. sin now, I sha-11 sin later, I 
would sm now, but I can*t." This is one tendering of the vifew 
that heaven is that perfected^ existence m vhich want3 preseVt, 



but now derfiedi, will Tje-.-satisf iea • «. 

_,Al^^^>^is]_^gqesW, iDZ^Qurse^^-.tha-^--t4ve-z-p£ob^m-r<>f — ' 



optim'aluty—therefora tft6 need fol: policy — would be' banished from 

^ _ • ' * 

any worlff in which human wants or desires are perfectly balanced 

by -their satisfactions. What else could paradise be except^ a 
' ^ t , * • ^ ' ^ 

condition in which all human desires are satisfied?^ 

There are two genetal sti;ategies always sufficient to jjroduce 
such a-solu^^idn* The f irst lies on- the s^ide^of-i^oing something 
about tfle satisfactions available, to human beings, (productivity), 
and the second lies, on the side of doing something*about their 
desiresMeducatiomand the development of character); The first^ 
leads us always to sol^e<^the problem of optimaiity by 'the 
.provision of abundance; there is presumably no problem of 
-satisfying* wants. There' is enough of everything, induing 
erfough justice and enough virtue^. When there- is no problerp of 
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LS no confl: 

no problem of optimality- 



satipfyin^ wants, there is no conflict of goods and/ therefore, 



The other-strategy is the converse. There is no scarcity of 
what nobody wants. So the second way to rescJlve the problem of 

^optimality lies not in the satisfaction of wants, but in thei'r 
•control^/ their composition. Thus, for Ghandi, diamonds and mink 
were plentiful, not because 'they were any the less scarce, but 
because they were not wanted. And not being >/anted, they were 
abundant. It heaven is that condition in which wants are 
satisf^i-^d, then there may be abundance in heaven not because^ 
goods are maximized, but' because wants are "composed". In 

' neither case do problems" of policy arise, and the reason they do 
not arise is that such jneWods of composing the goods of the 
world or of reconciling wants and satisfactions would render 
choice unnecessary. There is no conflict of goods in paradise. 
And the fact trfi^t^there is consequently no need for oolicy is 
part ot the proof that to formulate a policy question is to 
formulate the conflict of goods within which it is "nested". 

fiut why do we not imagine paradise Ifecontain interpersonal^ 
conflicts pf w^nts? The answer \s that to do so -would involVe 
either the judgment that wants are improperly controlled or that 
goods are insufficiently supplied. to satisfy them. Such a 



condition would introducre a proDlem. Paradise^ould no longer be 

a perfected existence^ It would contain the problem of 

\ ' ^ 

optimality, a kind of allocational defect. Paradise would 

*. ^ . ^ . . 

contain the problem of composing the most satisfactory 

combination of what goods do exist, and who should get them in 

what^^degree,. In shor t,« such;^ condition would introduce into 

paradise precisely^ those circumstances that create the n^ed'^for 

policy and that dictate the features of any well-fo^'rmed policy 

question — Hbat are the ggpds i^ conflict? What is their best 

possible adjustment?' How can we reach it? What are the 

trade-offs?*^ * ^ . \^ , ' 

The main point of this apparent digr(|SSion is that problems 

ofi policy are an immediate and direct reflection of some 

-immenHBly fundamental character istics of the world affd of human 



existence within ^it. They arise because the goods — not simply * 
the interests — that human beings seek to secure' in the world are 

'interdependent and do conflict, not all the timj^in every 
*rjespect, but all of the time m some, respects .* Only in paradise 
can we imagine that' all human goods are simultaneously in 
sufficient supply so -that there i.s no conflict m their 
alloca^tion. It is important to note that since knowledge is a 
certain kind of good, then the idea of paradise includes the 
assumption that there is sufficiency of knowledge. But paradise 
does not arise because there is an abundance of knowledge. 
Policy questions are not banished from paradise because our 
capacity to know is perfected. They are ^anished rather because^ 
there is either an^abundance of all qd^ds or because there is a 
suitable composition of desires. ' 
* This formulation, however, is not all thaft: is needed m * • 
expos^ing the presuppositions of policy questions. It deals' with 
tha presupposition of scarcity, but not with the presuppositiot^ 

--of int^rdepentrencg b'^tweyeh'^goodsT ' XT'lT^ufTicrenb to show that 

when scarcity is absent — as it !ft in heaven — then no policy 
* 

problems can arise. ^But the example J>resupposes that it makes 
conceptual sense to suppose that all human goods can exist in 
abundance simultaneously . Our imaginary hypothesis assumes that' 
a bMn<j[^n(;:e of some goodfl can always bp ,<;pr!tirpfi wihhnni- gn i f i rAnf, 
costs in others* » ; ^* 

The fact is, however, that human goo&s do conflict ^ such' a 
way that they cannot all be provided simultaneously in sufficient 
su^ly to satisfy human desires* The point is central* Human 
c^oQds do conflict so that the price of securing the abundance of 
some , is always fa;llure to, secure as much as we would liKe of some 
other, ^flpus, if we succeed in providing as much equal-ity as is 
wanted, we are u=hlikely to have as much liberty as is wanted. If 
persons develop as much tolerance for ambigiyLty as is wanted, we 
are unlikely to have as much courage as is wanted, and the price 
will sooner or later become apparent* 

The view then i« that human goods conflict (read "values" if 
you wish) not simply because they are in^short sug>ply, nor simply* 
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^'because int^rpecsonal preferences conflict^, but^ simply because 

they are not structurally consistent. They cannot; all be 

maximized, even in paradise, -It is a* familiar idea that human 

wants or human interests conflict. But View here is that 

human goods conflict. The fact that hJman interests conflict is 

what produces political problems, finding an a^^ustment between 

conflicting human interests. But the. fact that human goods 

- conflict is what produces policy questions, finding an ad^u^trnent 

between conflicting goods. Human goods continue to conflict^ even 

when human interests do not. Even when all are agreed on a 

. -> 

single predominating interest — vi^:tory in an all-out- war, for 
example--Tthere will remain policy problems. 

In other words, there are no unlimited goods in the worlds 
There. are no goods ^which, if provided ig great abundance , would 
not have the consequence of certain- other goods being m snort 
suppiy. The ultimarte solution to any problem of policy lis, 
^therefore, to be found only m paradise; that; is, only under 
conditions m which problems of policy are not so much solved' as 
they are simply non-existent. Such a state of affairs may, be 
ideal; and m that respect, it may be optimal. But it is not ^ 
Dossible, and Ax that rApect, it is not optimal at all. That 
the ultimate soluK^ory of all pdlicy problems, is to be found only ^ 



m paradise 4^ may be precisely the fact'TFTat g^ives rise to tBe^ 
cDnsi3tent and apparently iperlradicable human impulse to th4nV of 
social solutions to policy problems in utopiaft terms, ^ 

But Utopian thinking is defective not merely because^ it pays 
too little attention to feasibility. It is flawed more 
fundaroentally beca^|e Lt pays no attention at ail ^to 
^politics. ^ It i^ a fact of iarrge significance that'no well 
developed literary exposition of Utopia ever includes an account 
of politics. The central as^Kjmption always is that m Utippia the 
ne^ed 'balance between coniJlicti^ng hum^o goods is rersolved, 
RecOTSideration is not needed. Therefore , the introduction o'f 
pontics .irrto Utopia would be a threat to — not a p^rt of — the^ 
.good life\^ -In Utopia, problems of polic^ remain on 



extent that there remain problems of monitoring the societ 
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managing its affair^. ^Politics is replaced by administtation, , 
*and the role of evaluation, if^it exists at all, would be reduced 
to serving the ends of mahagemenJt. 

Policy questions do not arise at all in paradise. Seri<^us 
-ones do not arise in Utopia. But the 'teasons are. different, v The 
reason that^ serious ones do* not ar^se in Utopia is not that goods 
are abundant or that* desires are composed, but that the inherent 
conflict between goods^^ taken as resolved. All that remains is 
management. Not even the presumed Utop.ia of putting evaluators 



in charge would alter that resylt. ' ^ 
V We must note a related ^feature of -policy q'uestions 
generally. Like a reporter- filing his story for the evening 
edition, whoever will answer; a policy question^ in- the real 
worli, must do so within strict constraints of time. The 
* reporter files a story by aeadline, but .always witSi the knowledge 
that there will be another deadline, and the ^re&ent story can be 
amended by the next as e^nts change and further facts are 
revealed. Two points are discernible in this observation. The 
first is that policy qu^^ions generally are aliswered in 
anticipation that the answer will be ^revised. The second point 
is that they are the kirtids of questions that have to be answered 
on time, even though the information needed for the answer is not 



-o n tim e, — Bot h poi -R-t-s- ar-^s^- f -rcm t empor al co n s - tr axTTtsr t ^ut tney 

have different implications. 

The first implies simply that oolicies are impermanent. We 
expect them to change. Often, they are not ^even very durable* 
They are not ^supposed to t^. In that respect, policy questjfons 
are unlike constitutional gOestions, and they differ from moral ^ 
questions in Exactly the same respect. We do nbt^ expect persons 
to change /their moral principles^ — the constitution of their • 
character — <^ith great frequency. We do, however, expect, policy 
to changfe' with experience and with fair frequency. "Policy** 
implies "politics^ and '•pqlrty'*^ — a point jihat I 'shair copsider in ^ 
more detail^ later. We may note, .for* the raomerwt, however, tha\ i^ 
there is a practice whose improvement ypuld promise the largest ^ 
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marginal gams in the formation of policy, it would an 
improvement: m the pracvce of poiitiics, not evaluation. 

The second' point 'is equally vital, ^^f^means that 3ust as it 
is better for tne reporter , to file a story on time without all 
the facts than to get all the facts and file the story too late, 
so also^ It IS better, in the case o£ policy, to maKe a decision 
on timei box without all the facts, then it is to get all the 

j> 

facts and^make the decision too late. In the c^se of policy, 

__ * _ • 

decisions have to be niade always witnm large limits of 

uncertainty. Some fecuction m the degree of. jr.certainty will oe 

helpful, but the degree of reduction normally required ror 

"academic research is both improoat^le for policy decision snd 

would oft^n be undesirable -even .if it were not improoable , -^-^ ' 

In other words, crude data arryiving on time afe always oetter 
than refined data arriving too la/e. So it is acceptable, even 
fortunate, that the methods requ/tr^d fo^'policy decisio.n are 
crude, even thouah tne usual methods of isesearcn are necessarily 
refined. To answer a policy question, we need as much 
information as He can get. But **as ^ucn" as we can get" usually * 
turn$ out to De less than we could get if we had more time and, 
at tJ^e sacne time, more than can be used and more than will ma<e a 
dif f ereJic^e^ to tne decision. ^ Policy questions, m other words, 
afe aiway's'answefea ~rrr~the mTa"st~of ^^r\c^zax'f^fr'^c"^^^t^^~^ 
always a point beyond which n^sre information — however more 
excellent — will contribute l^^ttle to the reduction of jthat 
uncertainty and do nothing to alter the dir^tion of the 
decision. > . ' ' 

All this is simply another ^a^pect of the claim that policy 
questions are practical rather than , theoretical. They are 
questions of the sort that need to be answered^ and that will be 
answered, even when we do not. know which among alternative 
answers is the best. If these observations are credible, then it 
is possible to understand the claim sometimes advanced that 



*Note the offense involved in calling them "policies"! 



academic research is useful for policy decision inv^fifcly to its i 
excellence as academic research. ^ ' 

But there is another, ^nd more f a r— ^reaching , implication of 

, these last observations. Suppose we define the character '.of a 
professional — as I think we must — m relation to the degree to 
which the professional's practice requires accuracy of approach 

'in the midst of. uncertainty^ If we imagine a two-dimensional . 
space defined by (x) increasing uncertainty in the predictable 
behavior of the materials dealt with in any practice, and (y) 
increasing uncertainty in the consequences of one's actions, then 
we would be able to array the practice of professionals, 
craftsmen, art;ists, and tiechnicians as dispersed along the 
diagonal. . - ^ ^ 

I would suggest (and this ;s only conjecture) that such a 
definition of "professional'* would capture important f%$tures 
(though not all important feature*sj of what we ordinarily mean by 
the term. But such a definition has interesting and important 
consequences. It means, among ofcive^ things, that the education 
of professionals is an education in being able to make 3udgments 
and'decisions in the context of large uncertainties* Such 
'education, in a deep sense, is education not in tech/jical skill 
Inor withput tqchnical skill) , but in the capacity to deal with 

unce rtainty, to Ijve with doubt, to cfiange one 's min d. It is 

a^lVays ultimately h kind of education in self knowledge — learning 
one's limits^ , - ' 

'*3ut more importantly* and more concretely, .it would tend to 
rearrange our eonpeptions of the relation between the practices 
of ^cra^fts and professions. For eKample', it would mean that, the 
neuro-surgeon , although Assessed- of greater technical skill and 
manual dexter ity, , nevertheless^ deals jj^th a more predictable* 
problem than *the nurse, who, havrng to deal with the whole 



>p^tient, acfe in the midst larger ^ though often less seripus, 
uncertainties of both material and consequerices. In short, by 
such a View, the nurse turns out to be engagjed in a pfactice that 
is more pj^ofessional -than the surgeon*^. , 
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In an analogous way, tVe implication is that the evaluator," 
insofar as he presses for greater certainty and actually seeks to 
become the determiner of policy, is less of a professional, at 
h'ls ^est r than the politician or executive, at his best . The 
drive of evaluators, to whatever extent they seek to find in 
practice a means of resolving policy questions, if in fact the 
driVe to make the politician an ev-a lua tor ,<* which is to sa/ a 
techni?^ian*^of policy decision. Such an achievement, if ever 
realized, would copstitate th*e mast radical transformation of our 
political institutions and the practice *of policy decision that 
one can imagine. Even if it could happen — which is doubtful — it 
would be undesirable if carried very far. But th is, discussion 
carvnot be extended, refined, or made convincing without a further 
.set of distirsQtipna. 



, II 

I 

Facets of the, Policy Process » . 

r 

, Between policy analysis, policy forniat iori,-^^licy decision Or 

promulgation, and the poPitical analysis of policy there lie 

\ 

clear differences', and the practice of evaluation will relate 

i ' ■ . . 
differently to each. The tendency^- §xists to regard these four 



activities — analysis, formation ,• decisior?, and political ' ^ 
analysis — as steps in the policy process. But that view is 
•misleading, because these activities are never Eully aiscrete m 
practice and they do not occur in any persistent sequence. 
Nevertheless, there is a distinction of practice corresponding to' 
each activity, and'each practice, moreover, ha? its distinct kinci 
.of theory, - . ^ * 

Policy Analysis • ' ^ ' 

Policy analysis can 'be defined as the rational'or technical 
assessment of the net marginal trade-o*ffs between different 
policy choices. The quest^-on becomes "Sihich set of values*^i11 
be advanced, which wilTnOt, *and with what net benefits?** This 
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is the same kind of question that we confront, say/ in thfe 'design 
of a band drill, Wbat should be the design? The question is 
"nested" in a set of values. We want low cost, high safety, ease 
of handling, and durability. We can ask and rather precisely 
determine what marginal gains in one of these values will produce 
what corresponding costs in the others. If we "go for" greatest 
durability, then we are likely to get a higher cost and less ease 
of handling. If we "go for" the lowest cost possible, then we 
are likely to sacrifice something m the ^ay of durability and 
safety* The design problem is to discover a balance between 

0 

these' competing values. 

Enter the problem of incommensurabilities! Kow do ^we ^ 
determine whi,ch among the competing values is to %be given 
greatest wei-ght? ^^'hich has the^.gr eatest worth — low cost, safety, 
durability, or ease of handling? Not even the most refm'ed 
analysis of the costs and benefits will solve that problem. Such 
an analysis gives us the possibilities or a set at choices, but 

—it 'does not pick out any preferred answer from within the set. 

'Yet we need some procedure for doing 3ust that* In short, we 
n^ed a market decision , and getting-^a market decision is, no 
doubt, going to require a market analysrs . 

Is our^market made up of professionals? Or does' it consist 

=^ e4s«A^vt-iaily of -amat-e^r--s--and-4>ou^ehaW- e^^-tsmer^ I-f -itr-irs--the^ 

former*, thenjfthe groblem will probably be resolved on the side of 
durability and^ ^fety with a slightly h4gher pric^ If the 
latter, then * by all means, the decision will probably be to 
minimize cost and sacrifice durability and, to some degree, 
safety. But then again, the market decision mignt be to "go for" 
'Jr^e whoJLe range of the market. Produce a -variety of designs 
repCese'nting^ the full range of choices revealed by the^analytic 
exercise. Something for everybody.* . ^ , ' 

These activities are rougbl^ analogous to the distinctiq;^s I - 
want to, make in the case of policy. * Merely setting . forth the 
marginal qpsts and benefits of a range of chqices is one 
thing^-^pblicy analysis. Selecting one balanced choice oi^ *a range 
from within the possibilities is another thing — policy 
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formation. The decision as to which choice or choices will be 
made is still. a third — policy decision* And performing the 
market analysis needed for that decision -is yet a fourth — policy 
analysis. 

For example% sup^x^se we entertain the prospect of 
distributing educational assistance to students In preference to- 
institutions and' that we are resolved to do so on the'b^sis of 
financiNal need* In that casef^we require access ^to financial 
information, and not simply on groups, trends, or categories of 
persons, but on each actual individual. If, we propose this'kind 
of policy as more just t;,han other choices, then, m the name of 
justice, individuals will have to reveal personal information 
that may have been regarded oefore as privileged, t-^o values 
conflict. We extend -justice , but diminish, m some 'measure , 
privacy. To secure a definable gain m one, we pay a definable 
cost m the other. Policy analysis asks "What is the net 
marginal gain?** A truly refined policy analysis, which rarely 
exists, , would tell us bow much we are likely to gain in the 
advancera*ent of justice for some corresponding cost in privacy. 
Buti no such analysis, no matter how refined, will tell us whether 
it is worth it. In order to resolve that question, we need 
something corresponding to a market analysis and a "market 

-d^e-is-iOfVs — We -ne^--a--po2^i-td<^a4:— emalysis and -a poirtica-i — 

decision. ''Policy*' implies "polity*^ and "politics" just as "*good 
industrial' design" implies a structure for marketing analysis and 
marketing decision. 

But consider another exafnplS. A Congressman asks whether 
pass-through requirements for allocating Title I (ESEA) funds ^ 
should re'st on tests of eduSational need rather than economic, 
need,.. The answer comes back couched not id terms of "whether we 
shoula" bOJt m* terjns of "vhat happens if we do?" That's polxcyj^ 
analysis. In ^either case, the funds would go roughly to the sam^ 
school districts—but not quite. What's the margin of "not 
quite"? Is "not quite" "very much"?l^Is I't "enough to matter"? 

ev^n if it is "not much" would the change create incentives, 
fo^^^^^l districts to pay rt)ore att^ention to "educational need" 
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in answering allocational problems? And if so> then would the 
incentives be enough to make a difference? And if so, then. (here 
we are again) how much of a difference? That's policy analysis, 

'But policy analysis -does not^ and need not stop ther^. It 
can ask not simply what the net consequences would b^ of doing X, 
but what those net consequences would be compared to doing Y, 
where Y is either what we are doing. already or some third 
alternative.^^ Tfte question for policy analysis is x\ot whether 
doing X is a net improvement over doing Y — better than doing 
Y — but simply/ v^at are the jiet effects? Whether it is better to 
have a drill of low cost instead of high durability will not be 
determined simply from an analysis of the trade-offs. It 
requires' a marketing decision. Similarly, whether given the ~^ 
different Consequences, it is better to do X than Y m public 
policy will not be determined by a policy Analysis. It will be 
determined by a political decision resulting from a .political ' 
process involving a political analysis, % 
In short, "^licy analysis /I's^^at rational, technipai'^^^^ 
analytic performance in whichhfhe' centr^N5y£stloiv-»^ot whether 
X is a good thing to do, but simply what are the^ marginal effects 



of doing X, a*nd what are the marginal effects aP^ontrasted with 
doing something else instead? Hence,' policy analysis is simply . 
an activity whose theory is the theory of marginal utilities , It_ 
is, by all accounts, an- activity that consists in the exeraise of 
theoretical^ gather than practical, rationality. It assumes that 
the policy question is "nested" in a conflict of values present" 
as objective states of affair^ in the society. It is an activity 
in wbiph e?Valuato^r;5 may take a leading role prbvided that they do 
,not suppose they are-actually evaluating p61icy, as opposed to 
iBGxeiy tfecOrAirvg— -either in prospect or in retrospect — tfie 
consequences, of doing ^ ^j^Y,^^ 



Policy Formation 

>> ' " 

Policy formation , is an activity of a contrasting genrA 

Po*licy formation is that activity by which we seek to gain^ 

agreement on what form'a specif ic- policy can or will take, as 

opposed to what form it ought to take,. Not evenly the most 

refined policy analysis will we have actually formed a policy 

statement^* Indeed, policy analysts are not typically in a 

position to actually . formulate policy. ^ For the latter, we 

need to engage m conversation, persuasion, argument,, and m 

(seemingly) endless meetings with those who will actually pen the 

regulation, mark uj) the bill, establish the procedure-s, write bbe 

guidelines, etc. The theory of 'policy formation can then be 

discerned as one aspect of the theory of government management 

and rhetoric * At the Federal level, it usually turns out to be 

the theory of inter-agency pdlitics, "Don^t fight over turf; 

just, tak^e up space** is a rule for the conduct of policy 

formation, I include here the^ theory of rhetoric because clearly 

it makes a difference .what things are called. The same policy - 



.that under one name may' never 'see the light of day will, under 
another nanje, pass without objection. Calling i*t "school aid" 
may defeat *it; but calling the same thing **n'afeional defense" iffay, 
insure its acceptance. ."If it matters what you\call it, then 
call^it something that mattery" is another guiding rule in the 
theory ojE policy formation, 

Policy Debision 

- Policy decision can be described as the authoritative action 
of. some off,ice, administrative* or legislative, by which a line of 
action, for the moment' at least, is established. Policy decision 

is not so much an activity or process as it is a momentary end 

ft » ^ " * ' " ' 

point in the continuiR<| business of government. It is that end 

^ ' • t 

.point that is* some time s<? supposed by the naive to capture the 

• *^ < . ' 

entirety, of the policy -p'^ocess — as though making policy could be 

redudeji simply €o ,an act of will or the result of divination. 

The' ttveory of policy decision is> simply tne theory of the policy 
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itself * It is the political *and legal theory by which auth'ority 
is dist^r ibuted, obligations for decision are assigned throughout 
the structure of political institutions, and agents of authority 
are enjoined to act. . , ^ ^ 

Political Analysis ^, ^ ' . ^ 

Unlike policy analysis, political analysis is concerned not 
with determining the net benefits of a given godtse of action, 
but with their political weight, The aim is not so much to , 
determine the net social benefits of a particulaf policy, bu-t* to^ 
determine its constituency. -If policy analysis is concerned with 
establishing what is good/ then political analysis^ is concerned^' 
with estimating who will vote for it, whether the best thing to ^ 
do is the same ^s the best thin^ that can be done . Hence, the 
thepry of political analysis is the theory of political behavior . 

We may gather these thoughts togeth€[i> in a brief culminating 

* summary. The theory of policy analysis is the^ theory of marginal 
utilities,. It establishes the set of policy choices, The 

* theory of policy formation is the theory of inter-agency 
politics* It is the governmental process by which a course of 

faction comes to be s^^cted. The theory of policy decision is 
nothing less than the theory of the policy itself, an||kthe theory 



of political' analysis is tfie^ tfiebry o£ poHticar behavior . _ When 
we view all of 'these actiyities together, riot as discrete steps 
in the policy, process^ but as distinct facets of a , social 
process — not one feature predominating and»now another — then we 
can discern more clearly when the professional practice^ of 
evaluation ^it and what their relevance is tJo the creation, 
prcSniulgation and implementation of public policy, 

Evaluators and ^evalua,tion can contribute to each of these 
activities, but not to each in the same way*. For example, the 
rational-standards of policy analysis are the standards *of 
theoretical reason, but the rational standards of policy decision 
and politi^cal analysis are the standards of political judgment. 
These are practical activities. This difference may |?elp to\ 
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explain why it is that when the question, **Vihat should we do?" is 

-a — ^ . • ^ / ' 

given a policy analysis, we may get one 'answer, and when given, a 
political analysis or when rendered in a^Iicy decision, we* may 
get an entirely different answer'. In* shor^, the exercise of 
-political judgigent is a practical activity. It is also an 
evaluational activity. But the result of that activity. may 
di'ffer from or even contradict the results of policy apalysis. 
What we should do — even the best thing to-do — may turn out to be 
one thing by policy analysis and a very different thing when it 
comes to pQliti<^l dectsdon. 

The grofessio'nal evalua^or can c^atribute in the^ context of 
government, .but be will cpnt;:ibute to all * of thase activities'^ 
only to the extent .that tne evaluator oecomes aXso a. politician^ 
and a political advisor. ^ Consider for example, the case of 
policy farmati<)n. The .evaxuato^ , as professional, can- 
contrit>Qj:e, but that contribution ^will be most, substantial to « 
whatever ext^^nt he become^s.a stU/Jent of bur,eaucracy and a trusted 

" - • ' . ,^ * . y 

counselor^ to^ author itative J.eadjership, / " 

^ * • * *» . * 

^ So the dilen^a iS'^this. Each of these activities involves 

^ evaluaCjuon in'^scme' broad- sense of" th^ t^rio-^ Each involves 
evaluation* in^ thg^ sense, say, that buying a d'am^ra dpes. But 
•only^in the c^'Pe of policy analysis is the evaluator 's role, as 

-pi:of essi:onal, njTTdrlTjtEd^^tjy ttTe~TO^^d'r^ralce~^oir6tl^Tf"^T6^ The ' 
evaluator*', as evalucftbr ,- is Hkely to make a contribution only to 
the conduct of policy analysis. But in= government as elsewhere, 
the -possession of knowledg^ can^bring with it a certain kind of 
power. To "the extent that the evaluator goes beyond his 
professional practice and with superior knowledge also wrns the 
confidence of political leaders, exercises political juB'gment/ 

J ' s 

anii acquires the additional skills of a practice^ political 
.observer of the present bureaucracy and an untiertam^f uture, theft 
he will cc^tribiite to eve r y fac^t of the policy process* But in * 
doing so, he will also become iess an evaluator in any limited 
professional sense and more a political leader or*public 'servant 
in a quite old-£ashioned-and conventional sense. His main 
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characteristic w-ill not be the possession of technical skill. It 
will be the possess^^dn of civic virtue. 
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?OLLC£ AND EVALUATIOH: 
A CONCEPTUAL STUDY 



Appendix 



The six papers upon which this text was "dlVeloped are: 

1* Gordon Ascher . The Interaction -Between Policy and 
Evaluation Within Oregon Department of Education ^ 

2. Gerald W, Bracey, The ^Relationship Between .Evaluation " 

and Policy in the Virginia Department of Education , 
(Draft). 

3. David L, Donovan and Stanley A, Rumbaugh, The Interaction 

Between Policy and Evaluation Within thg Michigan 
Department of Education , (March 3, 1980 Draf tO . 

4. James Gpld. The Interaction of Policy and Evaluation 

' in State Departments of Education , (March 1980) 

5. Alfred Rasp, Jr.. Evaluation and Decision Making in 

Washington State r 

6. Paul Sandifer. ' The Interaction Between Policy and 

Evaluation Within the South Carolina Department of 
Education / (October, 1979). 

In the notes that follow, reference is made to these papers by 
the authors* last names.- Numerical references are to points 
within the text of the conceptual paper by T. GVeen* 

1. Bracey gives about as good a review, of the confusion 

surrounding the* term "policy** as I have seen. He, together 
with Gold and Sandifer, observes that policy is sometimes 
frame<J without "clear, procedural iijplications" and also tha^ 
procedures are sometimes viewed as policy* Ascher tenSs to 
see policy as *pretty much limited to authoritative rules of 
procedure, and nearly-, all of the authors r at one time or 
another, speak of policy as sanething akin to "personal 
policy of a person in authority".' 



2. 



It has not seemed to me fruitful to aspire *after too strict 
definition of "policy". But to see what' it^ me fins in the 
context of practice niight be (iseful. Hence, I have tried to 
avoid what would appear ^tb be a fruitless "academic" .exercise 
of little practical significance by. asking "What constitutes 
a policy question?" instead of "What is policy?" 

Do all policy questions have these features? I think they^ 
do, but the papers do not clearly reveal that fact. On the 
contrary, it is pretty difficult to fin^ any policy 
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questions really carefully formulated in these papers. What 
one ^inds, by implication, is a range of descriptions of 
technical and political problems surrounding some 
significant events -in the history of State Department 
activities. The values in conflict are almost never fully 
dravm out. But they can be discerned inductively at work >ri 
the narratives provided, 

for example, both Bracey and Gold remark, but in different 
ways, that when there is consensus, it can be decisive in 
resolving pblicy questions. But they also indicate that, in 
effect, this proviso, amounts to saying,' "If all people 
^gree on what is to be done, then it is no' longer a policy 
problem" ^Bracey, p, 30), The implication is that a 
conflict between values in which the policy issue resides is 
an essential feature witftout which such questions w<5uld not 
be serious, * • 

Sometimes we observe from the narrative* the points at which 
th^-^iested values" of^the polic^^ question begin to emerg^ 
through time.- (See Bracey, p, 45, wher« they begin to 
emerge, but do not ever take the shape of a well-formed 
policy question. See also Bracey, p,'47, .See also Donovan ^ 
and Rumbaagh, pp, 55-56, where it becomes apparent that the 
policy issues arise as larger numbers of goods are permitted * 
to enter in conflict through constitutional change and as 
their presence becomes more ♦evident. Again, the same is 
evddent in* Donovan and Rumbaugh on p, 64, and pp, 72-73,) 

This feature of policy questions generally is also expressed 
,in Gold, p. 84, It is the essential requirement that he 
lists there as Contextual variable if3 , , The same can be seen * 
in the South Carolina experience over legislation proposed 
•in 1977, As SanSifer tells it,' the policy '<5uestion, to the 
extent that there was one^ "was nested in conflicting values 
on the. part of legislative representatives< 

The experience referred to here is the experience over the 
eMt standards at the secondafry level in- Florida and the 
acCLount given by Bracey on identifying the "cut-off score" 
for the initial tests in Virginia, In that latter case,' 
consider the extent to which political' considei^ations would, 
and probably should, enter quite beyond any <^nSiderations 
of technical decision, and the ways in which the necessity 
of those considerations reflect values in conflict and 
define the poli^py question. In shorty Bracey describes 
it, the policy c[uestion was not really whether there would, 
be a cut-off score (or what it .would be) but how to strike 
the appropriate balance between conflicting valqes at i^sue 
in that decision, • The reSult'was initially a decision^ 
withoat data. There followed a readjustment of that/^ 
decision, and it ended up being 'Very different in ' • 

.appearance^ but not very different in consequences, I 
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suspect from what existed before. (See ai/^^^^^^c^^n^i far) A 
pp;^ 159-160.) 

Is there any evidence in^ these papers' of the assumption that 
people would agree on policy if 'they could only agree on 
goals? I am not sure. It is interesting, howevej;, tbat , 
those papers in which policy is seen to be most nearly 
associated with management, monitoring, and issues of 
compliance, (Donovan and Rumbaugh, Ascher, and Rasp) are 
also the -papers wi^Wjin which issues of policy are most 
explicitly perceived >^ issues of management and 
administrative guidari^eT" 

This observation running through the paper is clearly ' 
recognized, though not this explicitly, in Sandifer in his 
extended comments on the problems of -framing evaluations a^ 
though they were addressed to academics, (See Sandifer, 
pp. 155-156, and>again*p. 157.) But this point is also 
explicitly addressed in Bracey, 

In Bracey 's account of the program in Virginia, he provides 
clear instances in which policy decisions are made without 
data, with ttie suspicion that decisions might have been 
better with it, but with another suspicion that they^'might 
not be different in either case. StiJ-l,.the view prevails 
both in Bracey anS in Donovan and Rumbaugh .that tlnr 
possession of such data not only makes decisions more * » 
rationally persuasive, but may be necessary because of 
statutory requirements and for allocational decisions even ' 
when it does not produce a different result. 

This notion that a well-fotmed policy question always 
contains thes^ kinds of questions is not something that is 
well displayed in th^oapers. It y)uld be interesting to 
take either a truly semous policy question (such as is' 
implie<3 in, the personal goals of^ the *SPI. in Washington' or 
issues of remediation as displayed in the behavior of tfie 
system in Virginia) or 'a matter Of procedure as implied .by 
policy decisions in other of these papers/ and really * 
Examine what the policy question Is/ rather than/r^s is 
usually the case, consider' merely the result j>f-sa policy » 
decision and describe its^ operation/ * / >^ 

Can one find in these/ papers the -residue or* evidence of . 
Utopian thinking? Wt/at would it look, like? Welli for one * 
thing there Is in geperal 'a failure to take seriously lo 
these "papers the piffeSence or the need for a political 
process. Where the .^presence is a'cknowledge, %s in Bracey, • 
Gold/' and-Sandifer jchere is also, it, seems to me, a general 
failure to take that r<^e seriously. ' Exception would be the 
Virginia paper. Bpt Sandifer presents' an interesting -te'st 
.Of this.. On^.p. 158/ he explicitly acknowledges th4 
necessity of taking the political ^ftrocess into accou;it in . 
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trying .to understand the role of evaluation. But he regards 
such influences as "extejnal" (to the Department, I assume) 
and not as essential. The same ambiguity xs expressed on 
;p. 154. 

Bracey*s paper presents another interesting/ sensitive and 
sophisticated example of this kind of acknowledgement. But 
I wonderl would it be Bracey's opinion that things would be 
better on the whole if such "non-logical" behavior were not 
so influential? It would be the Utopian impulse to say 
"Yes" and stop there, I suspect that Bracey (also Gold and 
Sandifer) would go on to answer that question with a "Yes, 
but • • . ." * 

t think it' would be interesting to consider what features ot 
evaluation within the operations of government regard the 
preservation of politics as something of an intrusion. Why 
is it not the view of evaluators that, after all, the play 
of political, forces i_s the primary and only essential method 
of evaluation that we really have to preserve?' 

I believe ,that the answer to this question is that 
evaluators shar? a Itind of Utopian vision in which rational 
decisions replace* ^litical decisions. That is to say, the 
maintenance of politics is seen as an obstacle to the 
conduot of effective evaluation and an obstacle to making 
evaluation contributions to. social decisions. In short, 
'politics tend to be viewed as replaceable by management a^nd 
management ten'ds to be viewed as sanething that should be 
guided by evaluators*. 

The papers almost uniformly testify to the claim that the 
role of evaluation in state departments of education is to 
serve the' ends of management and to keep politicians out of 
trouble. {See Rasp especially, on the last point.) This " 
te^d^ to be viewed by evaluation ^^orists as a defect, but 
it is usually viewed^^y those in state departments as their 
normal, natural, and rightful role. ^ * 
• ^ 

All of the papers, with the possible' exception' of Ascher, 
cpmment on the stringent boundaries of t^me under which 
evaluators in state departments operate and the ways in 
which this fact marks a substantial^" contrast between what 
has to be done in the context of government and what can be 
don.e' in the academic settiog. Clearly, they are in 
agreement -that this is one pf the major differences that 
tends to make evaluation theory of little relevance to 
evaluation practice, at least as it occurs in Stat^ 
• governments. 

It is interesting, I think, that in none o'f*t±ese papers is 
there-^^y detailed story about framing a "better than" kind 
of judgment in the case of policy analysis. Rasp remarks 
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that the ^section m -Washington State never performs policy 
-analysis. The same could be said of Ascher's account. But 
'thi^s is probably an expression of the fact that well-formed ' 
policy questions seldom surface in the context of State 
educational policy.* That is, as I have observed already, 
the full explication of goods in conflict virtually never 
comes to the surface wij^in state governmental affairs. If 
It did, as a matter of course, then„ there would have^ to be 
stories about ''better than" kinds of judgments in the 
accounts given. ^^-^ 

Policy analysis necessarily involves, the study of marginal 
uti'lities and really excellent policy analysis (which seldom 
•occurs)^ would require a study of net marginal utilities. 
One step m that kind of analysis will always be simply 
"finding out v^at happened''? in other ^words , it requires the 
study of 'effects. Note: "Study of effects" is what nearly 
all experimental and virtually all intervention designs are 
about, Effects can be "greater or Less" more in one area 
than another, Sut tAe "study of , effects", though an 
essential step, will never add up to a study of wbr^. 
gives a good illustration of this when he describes^ie 
circumstances in Oregon by which any decisi^on of wor'th is 
"kicked downward" in the system. Studies of "effects", 
though essential as a step in policy analysis, remain 
nevertheless only a step. Bvaluators in state departments, 
contribute to the satisfaction of this step, but they 
'cannot, by that means alone, complete a policy analysis. 

None of the papers reflect the occupancy of a position * 
sufficient to determine the actual formulation of policy. 

The theory* of policy formation could be described as the 
theory we use to predict the behavior of political and 
bureaucratic leaders. All of the papers^ but especially 
Bracey and Rasp recount the significant dif f erence . that is 
created 'by the personalities-of i^ar^^ leaders. Evaluation - 
practice is substantially governed"T:^ the behavior of 
political leaders. So the theory of policy formation, as 
the theory of inter-agency politics, is likely to be 
-ejtfpressed in the descriptions that political officials give 
and the descriptions that their subordinates give of their 
ovn personal qualities, and their own personal aims as 
actors in the political arena. 

Notice that "political weight" is different from "rational 
we-ight;".y . , t 

The set of policy choices is also established by what are 
sometimes called "peremptory rules". (See Braybrooke and j 
Lindblodfn^/ A Strategy for Decision .) "Such rules tend to 
establi^sh the moral limits within* which policy can be 
selected, but, at t;he same token, they t6nd to guarantee 



that policies are to be chosen from among alternatives, all 
of which are morally permissible or have worth. In that 
sense, defining the set of policy choices is the expression 
>of moral conviction and value estimations, but selecting 
fran within that defined set is not. This is partly the ' 
reason why, with even the best; evaluation data, we are 
unlikely to arrive at policy choices that are substantially 
different frorfl those we. would arrive at without such data,' 
The range of choices i3 already subsjiantially set by 
considerations that; define the set of alternatives^ and 
those consider-ations, being peremptory, do^ not permit a very 
large range of differences to arise. 
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